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A This report by ther Comptroller General to the United ' 
States Congress discusses the review made of the Department of 
Educavtion f s National Diffusion Network,^ particularly its efforts to 
improve Title I reading programs through information dissemination on 
exemplary projectsv Chapter fcne examines the Department of *l 
Eatfcation f -s dissemination activities, administration pf Titled 
responsibilities and objectives, scope, and methodology of Title I 
projects. Chapter two discusses the^need to improve th<* effectiveness 
of 'existing programs through the increased and' more cost effective 
use of exemplary projects. Chapter three discusses the lagk of 
adequate evaluations* criteria for assessing project effectiveness, 
the "need to improve the monitoring of State Title I* activities, the • 
differences in local project evaluations, and* the need to improye 
network dat'a- on exemplary projects,. Following a concluding chapter 
are appendices whicjif discuss the^selection'^of school districts used 
' for site visits and analyze €he "Trading achievement gains, of Title I 
^students in thirteen school districts. The last appendix is a letter 
from the Afcting Assistant Secretary for Elementary -and Secondary 
Education in the Department of Education which .comments on a draft 
report of this document. (J.CD) ' * . 
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OF THE- UNITED STATES ' ■ . 




Grealer.Use Of Exemplary Education 
Programs Could Improve Education 
For Disadvantaged Children 



DC 



Although low-,abh1eving students in mariy ofy 
.the Nation's federally funded compensatory 
■ elementary and secondary education projects 
have improved their performance,-Jarge per- 
centages of students in some jjrojects continue 
to fall further behind iheir peers. 

The National Diffusion Network is the pri* 
mary' system the Department of Education 
uses in meeting the congressional mandate to 
fnsseminate information about successful edu- 
catioo practices in State and local education 
agencies to improve the quality and effective- 
ness of - F.ederah programs. Although some 
school, districts have adopted the Network's 
exemplary projects, *the number of adoptions 
is too small to greatly affect the overall qual- 
ity 'of the Nation's compensatory education 
projects. A greater effort should be made to 
identify- projects needing improvement, and 
better data on the effectiveness of the Net* 
wo&'s exemplary projects should be made 
available to school officials. 
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Tp the President of the Senate and the 

Speaker of the- House of Representatives ' . ' > , , 

* v • , 

This report discusses the potential for improving education 
for disadvantaged children through greater use of exemplary 
education ( programs available through the Department of Educa- 
tion^ tfationfcl Diffusion Netwprk. ' We examined the NetWork 1 s * 
efforts tpt promote the adoption of exemplary prggrams. by local 
school districts # tp improve reading projects conducted under 
titlb-I of the Elementary and Secondary Education kct. 

Copies of _the, report are*being sent to the' Director, Office 
of Management and Budget, and the Secretary of Education. 
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COMPTROLLER GENERAL--S 
REPORT TO THE CONGRESS 



'GREATER USE^V EXEMPLARY E^UCAT^ON 
PROGRAMS COULEI.XMP.ROVE EDUCATION 
FOR D I SAB VANTAGE D CHILDREN 



• * 



D I. G E S T , ! 

— i -- y ' - . 

Federal programs for elementary » and secondary edu- 
cation ^?e intended to help disadvantaged students 
acquire basic education skills, improve their rate 
of academic 'achievement, and ultimately help them 
•become self-sufficient and self-supporting. By far 
the Targes t,of these programs is <2tiat authorized ; 
under title I of the Elementary,, and Secondary 
Education Act. Title I was envisioned as, a potent 
tool for dealing with poverty and it* attendant 
conditions, su.cn as illiteracy, high dropout rates, 
'delinquency, and crime. (Seep. 1.) 

To improve the effectiveness of these programs, the 
CongrCss mandated that the Department of Education, 
(ED) -disseminate information about successful 
education practices to State and local education 
agencies. 

The" National Diffusion Network, established by ED 
in 1974, disseminates | information on various, ex- 
emplary" education projects that have been found 
^ highly effective i The Network is ED s primW . 
mechanism^for d'isseminating^^ormation on exem- 
S?ary title I projects and for helping local school 
districts to implement such projects. GAO reviewed 
, -the Network's 'performance in helping local -school 
". districts to improve their title l*m-,«n ta bv 
adopting exemplary projects. (See 

OPPORTUNITIES. FOR IMPROVING 
TITLE I BY ADOPTING [ , 
EXEMPLARY PROJECTS 

. Although many' title I reading projects areimproy-. 
ing the academic achievements of educationally dis- 
advantaged students, *ome projects are not success-, 
' ful and flhouM be improved. GAO s analysis of stu- 
dent progress in .about 340 schools in 14 
. reSealed 9 that many students continue to fall behind 
their normal (average) achieving peers. Title I 



^projects by 
pp. 2, 6, and 



services, are discontinued fyor 
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achievement. Furthermore, m*ny needy students are 
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^excludejd from receiving title I services because 
of the high cost of some projects and limitations 
on t£ie amount of available funds* (See pp. 8, 
9, li, and 12.) ^ 
— \ f 

"GAO's comparisons of the accomplishments* of 
title I projects wi th *the reported capabilities 
of thd 'Network* s exemplary .title I projects- 
.indicated th^ many school districts .could im- 
prove their title I Services >by adopting ex- 
emplary projects. Not only were the exemplary 
projects Reporting ''greater achievements, but 

J:hey sometimes cost l£ss and could ^erve more 

"students. (See pp. 9^to ll.) 

G RATER EFFORTS NEEDED* TO ' * 

ENCOURAGE ADOPTION^OF " 
EXEMPLARY TITL^ I "PRACTICES < 

Some school districts have adopted* exemplary ## 
projects for title I settings, but .the number of 
adoptions is too small to have any significant 
•impact on improving the effectiveness of title I 
in meeting. its r^tionwide goals. (See p. 17.) 

" Evaluation criteria and State 
monitoring need improvement 

4 

Title I administration has short c<&nings at the 
Federal, S,tate, and "local levelfe. . ED ha's not 
prescribed adequate criteria for S^ate ajid local 
title I officials to use in assessing their proj- 
ects,, arid Federal monitoring l>as been^weak and 
has failed to emphasise project quality. Sim- 
ilarly, State education agencies haVe provided i 
limited monitoring of local project results and v - 
haVe^not "ensured the adequacy of project evalua- 
tions and* the effectiveness of title I projects. 

« Without adequate Federal and State emphasis on 
• project quality, relatively few local scliool dis- 
tricts have sQji^ht to improve title I* projects. 
"Local title/ 1 officials generally hatfe not u^ed 
xpro jeet evaluations in project planning and im- 
provement ahd, income cases, have not developed 
local, evaluations that are suitable for identify- 
ing project weaknesses* (Se,e,pp. 17 and 18.) 



Information is needed 

on accomplishments — • 

of exemplary projects # s , 

Relatively little information is available "about 
the capabilities and accomplishments Qf the Net- 
work 1 b exemplary projects. TJierefdre^ local 
scho/61 district . officials are not convinced that 
exeirolary projects are better than their ^wn, arid ~ 
-few sfch^ol 'district^ have sought to implement ■ ^ 
these projects. (Sfere p. 28.) 

g * 
Increased »followup of project \ , 

-adoptions is heeded . . 

little is v known about the achievements of school 
districts which have adopted exemplary projects 
because the Network has made little .attempt to 
follow up d%jproject adoptions. ~ 

The Network's revised operating - instructions call 
for monitoring and evaluating project, adoptions- and 
for prpviding data on project outcomes at the adopr 
tion sites. The revised instructions, however * do 
not specif^ v 

-^the nature pf monitoring^ an4 evaluation to Be* . 
performed or . , ^ 

"—the data to be collected and reported on project 
'outcomes . ( See p , 30 , ) \ 

* 

Limited* data about results a lso* hampered Network 
officials in assessing the Network^ itryp^ct on. 
the title I. prpgram. .Because the Netwp/rk did 
not have uniform reporting requirements far its 
operations , it did not have reliable information 
on the number of exemplary project, adoptions 

that served title I projects." |S*ap. 33.) " 

* - ft 

' RECOMMENDATIONS , • . 

^ " ' < ' \ . * . ^ 

To improve the procedures" for identifying and 

, correcting title I project; weaknesses/ the • , 

•Secretary of Education should ^ • , 

✓—provide substantially increased guidance arid • 
technical assi^tajvqe to State and local % school 
officials in developing- criteria for assessing •* 

4 he effectiveness of title I projects? " ;> 
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~expahc( ED*~s~moKitoring efforts for assessing 
; "Whether State education agencies are (1) mfeeting 
/• ttteir responsibilities to evaluate the* ef fective- 
' 7 ness of local title- 'I projects and (2) helping 
local school. districts to improve their title I 
projects? - , ^ m m * ^ • ^ 

—direct St^tt^edtjcation agencies^o (3j assess * * 
the validity bf procedures local school dis- 
tricts use 1 to evaluate thel,r title I projects, 
(fc) prescribe 6orrective measures where pror 
ceaures are inadequate ,-^nd (3) h^lp local 
schOol^pf ficials use evaluations for detecting 
and correcting project weaknesses? *and 

^7- advise State educatiorf agencies to encourage 

* local school of'fj.cials to use Network assistance 
in impjrgj/ing title I projects.* - 

To qonyince potential adopters of the merits of 
exemplary projects, the Secretary should direct 
th4 Network to prdvide complete, current informal 
tiod about the capabilities and accomplishments 
of the Network's exemplary pr9jects, including 
(i) their impact on scholastic achievement, stu- 
dent behavior, and sustained growth, (2) their 
cost effectiveness, and (3) the results of past 
adoptions by other schools. * * 
-v 

To provide more accurate data on the voluijie of 
titl^ I adoptions an<i ensure th^t adoptions 
aVe successful, the Secretary should make sure 
^that th^ Network 1 s 4 plans to require followup oij 
project adoptions ar,e w adequate to (1) provide 
accurate data on the number of adoptions and the 
schools affected, (2) determine whether the 
projects Taave been installed correctly and are 
functioning properly* and (3) provide for addi-> 
tional implementation assistanqe wfjpre needed. 
(See pp. 36 ^nd 37.) 

• » * 
AGENCY COMMENTS 

In a draft of this report, GAO pfropose'd that ED 
prescjrlbe specific criteria for StAte atid local 
official^ to use in assessing the Effectiveness 
of title 7 projects. In its response to the - 
draft (see app. Ill), ED disagreed witfc GAO's 
proposal. ED believes that Its October* 12, 1979, 
title I evaluation regulations, which provide 
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-models and technical standards for evaluating^ 
title I projects, are adequate criteria on ( 
evaluation. * • ' * • " 

• / , . 
GAO believes that ED's obligation in fulfilling ltd 
requirements , does not endj^ith'the publication of 
evaluation models and technical standar&s. ED^shoulti 
place increased emphasis or* providing tGchnica], $s- 
sistahce at the State and local levels in developing 
criteria for assessing the effectiveness of title . I 
^projects. This would be consistent wii^h' ED 1 s long- 
standing policy of i^minterVention in the program- 
matic decisionmaking prpcess at tire State \and ;looal 
levels. ' . 

According to ED, implementation of the title I 
evaluation system Yfill permit local school dis- 
tricts* to assess their title I achievement gains 
on the .same metric or scale and therefore permit 
States to review results across school districts. 
ED sa j.d that National aggregation will enable State 
and local education agencie^ to view the levels of 
their gains in light of the ranges of tfains re- 
ported nationally. "Additionally , ED should^ use the 
national, regional, State, and district level infor- 
mation that It collects to provide increased guid- 
ance to* State and local officials concerning^ cri- 
teria^ for assessing the effectiveness of title I 
projects; GAO has revised its recommendation ac- 
cordingly. ED agreed with the thrust of' GAO' s 
other recommendations. ED's comments are discussed 
pji^pkges 37 through 42>. 
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PTER 1, 
INTRODUCTION 



Tq improve the quality and effectiveness of Federal programs 
to assist educationally disadvantaged children, 4 the Congress man- 
dated' that the Department o£ Education*" ( ED) identify successful ^ . 
education programs -and disseminate information aboi^t them to ^ 
States ancl local school systems. To meet this mandate, ED H*s^s 
the National Diffusion Network as* its principal system to dissemi- 
nate information about exemplary education projects and to help 
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schocirLs implement such projects. v i f 

* y 
Recognition of the najbional need to assure that educationally 

disadvantaged children fire provided suitable educational Opportuni- 
ties has, spawned a number of Federal programs to assist education • 
in elementary and_ secondary schools. Federal programs 'for elemen- 
tary and secondary educat£9n generally qjre intended to help the 
disadvantaged to develop their fcasic skins, improve their ratef 
of academic Achievement, arid help them become self-suf f icidnt and 
s^lf-supporting. * % < 

%The largest .Federal program for feiementary. and "Secondary 
education comes under title I of the Elementary and Secondary 
Education Act" of 1965 (EJ3EA) (20 U.S.C, 241a). Tlje Congress, 

• repognizing the link between ^inadequate educational opportunities 
and poverty, authorized, tfie title I 'program to assist educationally 

5disadvantagfe^ children. Title I was envisioned as a potent tool 
' for de'aling with poverty and its attendant conditiqms^ such as 
illiteracy, high dropbut rates, delinquency, unemployment, and 
crime. ^ * • " * 

M 

m m 9 Under, title I', funds are provided to States and ^ocal school 
districts to help them provide compensatory education programs for 
educationally disadvantaged youths* The^se programs focus. on de- 
veloping basic skills in reading, language 'arts, and mathematics; 
Since the program's inception'in 1965,* funding has totaled about 
-$30 billion. For fiscal year*1981 ^>but $3.7 billion has been' 
nrbvided for the program, yhich is 'expected to reach "about 14,000 
of the Nation'^ approximately 16,000 sohool' districts. * 1 v 

In its desire to improve £he quality of the Federal education 
programs, the Congress recognize^ the need fl»r dissemination of 
effective educational .practices. Cpng'resSional mandates to-'dis- . 
seminate information to improve education programs now appear in 
various lawfe. Title I of ESEA mandates that m ED (.1) develop proce- 

* dures for identifying successful educational-projects and # practiced 
and«(2) disseminate' information on sucJh^ pro jects and practices to 
States and loca'l school districts. , 



. ED DISSEMINATION- ACTIVITIES .• * 

The National diffusion Network—the largest o£ ED's 
dissemination mechanisms — is responsible for <!%seminating in T 
formation on a wide -range of education pr^jects'in such diverse 
areas^aS - /bilingual, and immigrant education, career and vocational 
educatiop', early childhood education, environmental education v 
special education for 'the .handicapped, arts and communication 
slyLlls, mathematics, reading, and language drts. The Network is 
also J5D!&.prijnary means* for disseminating information on exeifiplary 
title i projects and practices. , ^ t 

# Sinbe its inception inll974^ the Network funding has totaled . 
about $59 million* As shown in the graph on the following page, 
funding has been somewhat ei^atic, averaging about $7*4 million 
per year but ranging from "zero in 1976 to JM1.5 million in 1979. 
Of the Network funds, ED officials report that about $1.25 to * « 
H $1.75\ million per year is, directed^ to title^I dissemination ac- 
tivities. By comparison, the annual title I budget now exceeds 
$3 ^billion, % \ , ' m ' r> 

Before fiscal year 1976, the Network was funded under 
t^itle III, section 306, of ESEA, as amended. - This legislation 
eltpired at the end of fiscal year 1975, and the Network was ngt ^ 
funded in fiscal year 1976. In fwal year 1977, the Network was 
'funded under se'ction 422a of the <»neral Education Provisions Act. 
Beginning with fiscal year 1980, authority for Network funding , was 
contained in sections 303«and 376 of ESEA. y . * 

^Pfie^Networ^ goal', $ts„prescribed by ED, is to im- f 

prove education practices by broadcasting, information about 
successful projects and practices and assisting in the widespread 
implementation of these projects and practices. ED instructs J 
Federal and State education program officers to be alert to * < 
identify highly successful .education projects, including titlfe I s 
projects, and to encourage the project developers to seek valida- 
tion by the joint Dissemination Review Panel. / , ^ 

* t 

E£> established tTie J6int Dissemination Review Panei to review 
data on educational projects and assess whether -the projects had 
produced substantial, educational improvement and whether they 
could producj| similar improvements at other locations.' Officers 
of the various ED program divisions may nominate identified proj- 
ects for panel review. % Projects approved by the" panel are termed 
11 exemplary" —they are the only, ones that may be endorsed c^nd dis- 
seminated by the ^Network. 1 . . 

'The Network's activities to disseminate information about 
exemplary projects and to help schools implement the projects arfe 
carried out largely by education project "developer-demopstratbrs" 
and by dissemination ,f facilitators • 11 - 
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DeveloRer-demonstrators v 

School distr^qts that have exemplary projects approved by 
the jfeint Dissemination Review Panel can apply for developer- 
demonstrator fundg, from the Network* Developer-*demonsfcrators are 
typically education agencies that originated* the* pro jects and are 
responsible for helping school districts implement* their* projects.^ 
The developer '■demonstrators are expected to provide information* 
describing their projects, develop materials needed to, implement 
tfteir projects, demonstrate their projects to interested fchopl 
officials, 4nd .Conduct -training sessions for ^chool personnel." 



H Not all exemplary projects are selected to become developer- 

demonstrators. ' In selecting deveioper'-demonstrators, Network t 1 
officials attempt to acquire a wide variety of projects and educa- 
tional approaches for school systems to choose from. By fiscal 
'year 1979, the Network hffd awarded contracts to* 21 developer- 
demonstrators* to market and help install exemplary title I projects. 

State f aciij^tdtors ' 

' * The Network also*contracts with organizations in specific 
geographical areas throughout the Nation to facilitate the adoption 
process by linking school districts with developer-demonstrators. 
Thpse facilitators usually are assigned to cover • specif ic States^, 
< and are called "State facilitators. 11 

, . . . > ' 

A State. facilitator ' s. primary responsibility is to widely 
disseminate information to loca> school systems within its State 
or area. Facilitators acquaint schpols with the exemplary proj- 
ects, help schools determine how the exemplary programs might 
benefit the schools, and provide a link between the -developer- 
demonstrators and schools that express interest in adopting 
^exemplaj^y projects. \ 

Network accomplishments^ , 

* * * * 

ED's assessments- of %he Network have generally been favor- 
'able^ based on observations such" as the following: 

— The Network Tiaa ^uqcessfully promoted a large number of 
adoptions? ED 'reported 5,600 "instances of adoption" 
since the/Network began in 1974. * 

— Thfe Network provide^ draining in exemplary project opera- 
tions to about 25,0C^0 teachers in 4,20Q schools during the 
1978-79 school yea^. f 

— The adopted programs are highly regarded by tii& adopting 
school official 7 ^. * 
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-«-The demand for. the Network! 8 services exceeded its capacity * 
. ' to provide them. / * • # • 

— Existing exemplary proglams can be" installed .for aa average \ 
cost of §4,000 whereas developing such a program scosts ^ 
.£4'00',0Q0. 

These assessments w^re centered on the Network's overall ac- yr 
' tivities in promoting program adoptions. The assessments did not 
-•distinguish between different types. "of education programs,, such as 
title-I projects. <* * ^ - m — 

* ADMSNISTSATION OF TITL^tF^ RESPONSIBILITIES. " " . % - 



* * Besides requiring 'that exemplary titLe I projects be JLdenti** 
fied and dlssefminated to States and local school districts^ ESEA 
requires "ED to J 

--review each State's title I grant applications and assure 
* P r °9 raI ^ P lan3, are adequate and/complete, 

^ /* —monitor each State *b title I project activities , for com- 
pliance with legislative requirements, and 

-^provide standard^ and models for project evaluations and 
provide, technical evaluation assistance *t.o States. ^ 

The Office of Dissejnination and Professional Development and _ m , 
.the Office of Compensatory Education have major roles in carrying 
out ED/s title I responsibilities. The Ofrfcee"of Dissemination and 
Professional .Development operates the Network, which is responsible 

for ' % 

— improving the Quality of education nationwide, including 
title I projects; - 

— disseri?Lnating information, on successful title I education 
practices; ' 

s * 

--promoting -widespread adoption of successful title I educa- 
tion practices;; - . 

— helping schools ta: inclement exemplary education projects; 

--providing training and technical assistance to adoptrn| 
, - y . schools,- * 4 ^ /.* " , a , * y 

— evaluating the effectiveness of ^the' adoption process; 
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— providing postadoptiop technical assistance to adopting 
schools; and • * ^ 



selecting dissemination facilitators' and developer- 
demons tutors to carry oi*t the Network' s activities. 

Although actively promoting £he improvement of title I proj- 
ects "through* the adoption of exemplary projects, the Network does 
not have any formai responsibility for , conducting the title I 
program. 1 The Office of Compensatory Education is responsible for 
the overall administration of the program/ which includes* 

— providing guidance to State and local school officials, 

— reviewing State title I grant applications, 
* • 

, Ik ^ 

! — assuring* that State title I responsibilities are 
fulfilled, 

yj . 
M — reviewing* and assessing title ,1 program data submitted 
by States, and # > 



— identifying successful education -programs and practices. 

OBJECTIVES, SCOPE, Afep METHODOLOGY 

Because of .the concerns of educators and evaluators that 
ti\le I projects are not achieving intended goals and the explicit 
, intent o£ the Congress that ED and school officials disseminate 
, successful practices to impr3y§"*title I projects', our review 
fbeusgd on the Network's ef^rts to inprove title 1 reading proj-' 
ects/^p;cJw^rd £his end, y/s. reviewed (1) the Network 1 s, actions 'to 
promote adoptions of exemplary, projects for f< title I, "(2) the suit- 
ability o£ data that school ^districts have^f or assessing the need . 
for title' I project improvement, an<f^<3) school officials 1 actions 
to t adopt better title I projects. Our :review focused on reading 
projects because most title I 'projects are reading-language 



oriented* , 
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Our review included examinations of legislation; Federal - 
regulations; Network ^records, reports, and materials; State and j 
loc^Jr education agency procedures, records, and project , assess- _ 
ments;. and* federally funded evaluations of title I and other 
Education programs ♦ We interviewed of f icials 'at 
* « + 

— ED's Office of Dissemination and Professional development, 
its Office of Compensatory Education, and the National 
Institute of Education; v 1 

— State education agencies Mn £ive States;, " 



• / 



— dissemination facilitators dn five States under contract 
with the Network; 

— seven exemplary project developer-demonstrators; and f 

*^ ' , 

* , — twenty school districts that had implemented exemplary 
title, I projects ♦ * * * ^ 

• " * 

Al'so, we visited. 47 local school districts in six States, 
.including the above/five States, to ascertain the Network's impact 

oh the districts. These school districts were % selected to provide 

a mix of small, medium, and large districts in'rural .and- urban . 
'locations. v The achool' districts we visited^re not > .statistically, 

valid representation of "all school distrifcts nationwide, * nor , * in 
'our opinion, was the sanple necessarily nationally representative. 

However, these districts represent 'a wide variety of sizes, types, 
.locations,- and environments. 

Chapter 2 of this freport assesses the potential for; improving 
title I projects. Chapter 3 examines the effectiveness of mecha- 
nisms to identify title I. projects needing improvement and of the 4 
Network's activities to, promote adoptions of exemplary projects 
for titl£ I.* Our conclusions 1 ahd recommendations and-ED's response 
are in chapter 4. ' t 
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• r ( CHAPTER 2 • 
GREATER USE OF EXEMPLARY PROJECTS- 
COULD IMPROVE TITLE* I PROGRAMS . 



( 



Low-achieving students in many of the Nation's title I projects 
have improved their reading id^Levemejit" rates. Some tit^e I proj- 
ects, however, are fnQt producing significant achievement *in reading 
and need improvement. Students in these latter tittle, Iypro jects 
are continuing to fail farther behind average achieving students^ ^ "* 
Often title I services terminate before students reacn suitable 
achievement levels, » Ip some cases, ttye highicost of title I 'proj- 
ects contributed to the premature terminatiomLof the services. 

Through the Network, ED dilsserninates information oft exemplary 
pro jects* proven capable of producing significant educational 
achievements that are worthy of implementation in other schools 
and sometimes cost less than existing t* tle 1 projects. Qur com- 
parisons of ^exemplary projects with title I,projects, in school 
districts^ we visited indicate that substantial improvements might 
be realized thrptffh. greater use,of exemplary projects. Implement- 
ing an* exemplary project in an Adopting school district, however, 
i*s a complicated propess requiring staff training- and assist^fice. 

NEED TO IMPROVE EFFECTIVENESS . .\ -. <^ 

OF EXISTING PROGRAMS r ) 

• — — ... . • ' / . < 

Studie^ of the. national impact of ♦ title I programs have raised 
questions as to whether the projects have improved students 1 
achievement rates. A 1975 ED-funded stu'4y }J showed that, on the 
, average, title I students tended not. t6 fail further behind non- 
title I students during the agaddmic yj§£Jv However, the study also* 
showed that many schools failea in theiar goals to help disadvan- 
taged students gain on their peers aftd Jtha^ many students were 1 
coirtinuing to fall behind. £D, reported' in its 1973 annuat evalua- 
tion report that title I pro jectsJiave demonstrate^ modest success, 
but that much work remain^ • « 

A 1980 study funded by ED 2/ compared th^, reading growth of 
title,, I students- with that of SxHlilarly disadvantaged students who 
did ngt hayq^any compensatory reading assistance • This study con- 
cluded that vitle I students (1) in grades 1, 2, and 3 grew .at a 
faster rate than disadvantaged students not receiving coirtpensatory 



* ' /J A 

l/ M A Descriptive and Analytic Study of ^compensatory, Reading 
Programs" (Trismen, Waller, and Wilder), 1975. 

2/"¥he Sustaining Effects Study: An Intejrim Repor€, 11 x/T^Carter /f^" 
October 1980 . " • i • • 



education* and (2) in grades 4, 5, and 6 grew .at the "same rate as 
the unserved disadvantaged students. ^ :# 

» ^ * * m t 

Using a different comparison standard, 4 w6 arial„yzed the reading 
achievement gains' of titl^ I students- in about 340 schools involv- 
ing 14f schoo.l. districts, (See app. II for analysis methodology.) 1/ 
, « Our analysis used a more stringent standard in 1:ha,t it corffpared the 
^pretest to posttes£ 'gain of title I- students with ' students at the ^ 
same grade level who scored at the national pretest average • We 
maffe this comparison to ascertain whether title I projects were en- 
abl-ing the disadvantaged students tp gain on th.eir norpaal (average) " 
•achieving peers. 2/ This analysis disclosed that many students 
* made significant gains While in tftrle I project^ For example, the 
reading achievement gains of 60 percent or more of the students in ' ^ 
four districts consistently equaled or exceeded T year per year^of 1 / 
instruction. However, in some other, districts, significant 'portions ,\ 
of the students continued to faLl behind the regular students. In 
about half of the years analyzed, 40 to 67 percent of the students 
were falling further behind their normal achievingSpeers, At the 
timg their title I aaaiatanee was terminated^, therstudeuLs in our 
sample were an average of 1*5 years below cfcade ••levefT" 

ItfCRE&SED USE OF EXEMPLARY PROJECTS COULD 
-•IMPROVE EFFECTIVENESS OF TITLE I PROJECTS 

\ > ; : ■■ \ 

To determine whether exemplary projects could mgtJce title I 
projects more effective, we sought to examine whether exemplary 
projects produce greater achievements than other title I projects 
and whether Adopting an exemplary project enables a title I project 
' " ' to improve its achievement levels • * r* 

Although comparable achievement data weife not always avail- 
. able for title I exemplary projects, our arvalygis of data, avail- 
able for six projects 3/ indicated that exemplary projects have 

•> * r . * ' , i. 

l/ED e1^>ressed concern about our reliance oh the grade equivalent 
" metric &r scale in this report. According to a repfort on con- 
troversies in evaluating compensatpry education prebared by the 
American -Institute for Research for the Motional Institute of 
Education, all available metrfcqs or scales have weaknesses. 
HoweWr, the report notes' that noffe of the available metrics or 
scaled has the same clarity and simplicity of meaning £s grade 
equivalents. ,0iir usd of grade equivalents is furthef discussed 
in 1 appendix II* * 4 

i 

\ . 2/The evaluation models developed by ED -compare the performance. 
' *■ of title I stupents with similarly disadvantaged students that * 
received' no compensatory assistance* 

■ * «t * 

3/A11 six exemplary projeqts were either developed for $itle I 
^projects or recommended by thfe Network for use in title I. 

— * £ { • • i , ' 
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good po£ential for improving the effectiveness of compensatory 
education in public schools. One tltle-X- exemplary project, for 
example, ''reported that 85 percent of its s&jdents gain, at a rate 
of more than J, year per yea'r* of Instruction • . For compatison, we 
selected 22 sampl,e groups of- title I studet^ts^in the. 14 schoql 
districts where we analyzed title F achievement. (See app. II for 
a description, Q f the' sample group selection grocess. ) ^^tudefljjits / in 



21 of the 2:2 sajnple groups were not able to* attain the i£ate of 
achievement reported by the* exemplary project £n afty of fthe^ years 
analyzed. . 1/ 



None 'of the^six exempla w ry projects reoorted average gain . 
rates below the normal achievement rate of > year's gain per year 
of instruction expected £or average students. In contrast, 'over 
half of the 22 sample -groups were below the normal achievement 
rate in at least 1 of the years analyzed and 2 groups were below 
this rate in each __year analyzed. One title i_ exemplary project 
reported that the mean gain, of students swerved was* 1*7 years per . 
year of instruction. This rate was not reached by 15 of the 
22 sample groups in any year analyzed. No sample group averaged 
this rate in eafdh year analyzed-. 

Other exemplary projects alSo reported significant accomplish- 
ments. One project reported average gains ranging from 1.4 to 
1*9 years per year of instruction for various sampLe groups. ,A 
project that used student tutors to assist^title I students re- 
poirted gains ranging between 1.2 and 3*7 years per yeafr of in- v, 
str.uctibn for botji the tutors and the students bteing tfitored % 
Another exemplary project reported that it doubled the. rate of 

'* gain of tlie students it served," ~ . m *i. 

* » 

According to ED, t:he Joint Dissemination Review Panel has 
. assessed the ^reported exemplary project gains and can attest -to 
their validity and educational significance. Information accumu- 
lated by one exemplary project showed that adopting districts had 
also achieved success* This exempl ary project, which regularly 
provided followup assistance to adopllng schools reported that 
achievement gains' of gtjudents in 21 adopting school districts 
rariged from !•»! years in gxades 2 and 3 to 1.8 years in grades 7 
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USING^ EXEMPLARY PROJECTS s 
COULD BE LESS COSTLY 



r 

4 



Many school districts have title I projects that cost more 
than some relatively inexpensive exemplary project^. . The ^gpmpen- 
satory* education projects in tihe 14 districts Where we, assessed 
achievement gains cost up to/$778 *per. pupil # per year. The average 
•per p^pilcost was $450 per year, and only one district had a per 
pupil cost below $300 per year* * 
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In contrast, the six exemplary 'pro jects mentioned above were 
much ies£ expensive. The most expensive exemplary project cost 
$281 per pupil per year, and the averag^ cost of the six prpjects 
was $180 per pupil, less than half tWe average c6sjr of the projects 
irt the 14 school districts. Thus, these exemplary projects were 
not only more effective in raising student achievement, but als;o 
less costly. Increased use of such projects might enable school 
districts to serve more students over longer peridds. • 

According to ED officials, millionsN of children eligible for 
title I services are not being served because of limitations in 
program funding. Additionally/ t% each year, title I services to a 
million or more children are "terminated prematurely (i.e., before . 
the studeftts reach normal achievement levels for their agfe and 
grade level). However, ED has not' assessed, the pptential for ad- 
dressing this problem through development and replication of fexem- 
nCary projects that cost less than existing title I projects.' ED 
Officials pointed out that project costs pan vary widely because 
of teacher salary differentials and other factors, such as the 
degree of ^reliance on low-cost tutoring approaches to compensatory 
education-, • * ' * * 

' v - - 

Eligible children not served ^ * * 

■ _ — ■ ■ # , 

According to ED officials, precise information ijp not avail- 
able on -the number of 'low-^achieving students eligible for title I 
services but not receiving* 1 ^ em. ED officials f estimptfe% .however, 
that "bnly aboufhalf of - tafr c S Nation 1 s low-achieving/^ligible stu- 
dents receive title I services. A 1978 study fozt-ED 1/ reported ' 
tfcat most of the # Nation's l&w-achievijpg elementally students do r(ot 
receive compensatory* education from title I or anw other source. 
Other studies »have reported* that compensatory education beyond the 
elementary leVel is almost* nonexistent. * 

Only one^of the five State education agencies we visite<T had > 
any information on the number of eligible students not served by > 
title I projects:.* -This State reported that about 219,000, or * 
65 percent/ of the^State's eligible -students were not receiving • - 
tittle I seryices. . v 

* * ' ' ( 

Services terminated 'prematurely ^ & 

r ' * 

_ -- -Title I services are discontinued fdr many students while « 

they are" still far below thleir peers >tv ^cademic achievement. 

This generally occurs when the students are advanced to a gracle 

level not having title 1 seri^ices or are replaced by needier ^ *. - 

students. %' 1 f 



^/"Students 1 Economic $nd Educational Sta£us % and Selection for/. 
Compensatory Education," V* Breglio, e*t al.; 1978. 
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ED officials testified %/ that many eligible students are 
excluded from title I proj r 6dts because sufficient funds are not 
available. According^ to EIr, most titiH^I services are at the 
elementary level j^and a hifh percentage of titled funds are 
used for the first three grades. ED -officials estimate, that an 
additional one-third to one-half of the • elementary stuotents need 
compensatory services and/'that the need at Jthe secondary level is 
far greater/ ^ 

{ f " ' ' 

£ducaturs we talked' to in the five States we visited ex- 
pressed ponceirn that tt(e need for compensatory education was not 
being met beyond the eajrly grades. The Congress/ concerned about 
the high rate of illiteracy, stipulated 2/ tjaat school districts' 
should consider extending title I services in intermediate and 
secondary levels^to sustain the gains of earlier service?. 4 Recog 
nizing the n$ed for compensatory programs at the secondary level, 
ED has begun to publicize exemplary programs appropriate for 
secondary schools. 

The results of pur analysis*of student achievement gains 
suggest that concern over the progress pf low-achievihg students 
denied compensatory assistance may be wtll founded. Nearly 
11,000 students in our sample groups hacK their compensatory 
assistance 'terminated or . substantially interrupted (by 1 school 
year or more) duringSjje period analyzed., When the assistance 
was terminated, the Students 1 .rate pf ^achievement gain declined 
by* an average of 0.6 yeafrs per year of instruction. While re-* 
ceiving compensatory a^ssistancs, most*6f these students were 
gaining on th£ir normal achieving classmates. ^However, when the 
assistance was terminated, the average rate* of gain dropped well 
fcelow the normal rate. Thia pattern of pronounced decline in 
rate of achievement occurred in ll^Of the 14 school districts 
analysed. ' 

, This decline in average achievement gain rate was confirmed- 
b£ a corresponding drop in the number and 'percentage of students 
gaining at or above* the normal rate of 1 'year* per year of instruc 
tion. As, shown in. the following graph, the percentage of stu- 
dents gaining at or above the normal rate for average students 
while receiving compensatory reading assistance declined from 
77 "to 41 percent when assistance was germinated. 



l/Hearings befpre a subcommittee pf the House Committee on 
Appropriations, March 15, 1978, p. 86. 

^/Senate Report No. 95-856, May 15, 1978, -Education Amendments 
of 1978.;. • , *' - 
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WHILE TITLE I 
SERVICES WERE 
BEING RECEIVED 



AFTER/TITtE I 
/SERVICES WERE 
TERMINATED. 
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Similarly , the percentage of students rapidly falling behind 
their peers (those gaining at half tl^e ribrmal rate or' less) in- 
creased f jronu 14 'to 38? percent when assistance was terminated. * 
This increase is shown in the followihg graph. 
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' PERCENT OF STUDENTS FAILING TO GAIN MpRE 
THAN 1/2 YEAR PER YEAR OF INSTRUCTION 




WHILE TITLE I 
SERVICES WERE 
BEING RECEIVED 



AFTER TITLE I 
SERVICES WERE 
TEFtMfNATED 



Compensatory reading students in the 14 districts we analyzed 
were still far behind * their normal achieving classmates' when their 
compensatory .assistance ended. The average achievement .level of 
these students was 1.5 years below grade level when thfir title I 
assistance ended. This deficit ranged from 0.5 to. 2. 4/ years below 
grade level in \tie districts analyzed. • 



ERIC 



14 



. 2$ 



In one large school district, title I* assistance was resumed 
for 1/lyS students after «they had been but of the program for at 
least . 1 ^school year. Their rates of achievement gaJLn dropped sig- 
nificantly when the title I, assistance was initially terminated, 
and vejryVfew were aBle to gain at the normal, rate or better while 
out: ,of tile program; However, when title I assistance was* resumed, 
the achievement rates of many of £hese students increased signifi- 
cantly. '/k^Jshown in the following graph, the percentage of stu- 
dents keeping up with or gaining on their peers jumped from 6 per- 
cent whll^ out of the program.to 78 percent when assj.fft.ance resumed. 
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Terminating title I , services before the 'students^'^^aiii accept- 
able achievement levels appears to conflict with the title I goal 
of assisting educationally deprived studentship acquiring the fcasic 
skills fteeded to become self-sufficient in a competitive society* * 
Services for most title I students are discontinued in the elemen- 
tary grades, whereas most dropouts occur between grades 9 and 12. 
/Department of Health, Education, and Welfare reported in 1977 
t nationwide about one in four students fail to gradu^e from 
igh ^hool. ..Dropout rates for large city schools may be*higher. 
One large city reported that nearly half of its students drop out. 
In our opinion, the premature termination of compensatory education' 
services increases the chances that ^underachieving students will 
fall further behind their peers and* eventual]?/ drop out of school. 
If this ocpurs, title I is unlikely to achieve its gdals of assist- 
ing the disadvantaged to become self-sufficient and of reducing 
illiteracy jand poverty. 
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CHAPTER 3 

0 .* • * ■ 9 

■ LACK OF EFF ECTIVE TITLE ' I EVALUATIONS 

» '* 

AND LIMITED EXEMPLARY PROJECT DATA 

' ' > — * " ■ '. * " , ■ 

' HINDER ADOPTION OF NEgfPRK' PROJECTS 

Although many title I projects need improvement, relatively- 
few schools have adopted exemplary projects. Only about 60 (3 per- 
l cent) of the 1, 958 school districts in four of the five. States we 
visited had adopted an exemplary project for^use- in their title I 
projects by the start of the 1978-79 school-year. (The Network fa- 
cilitator for oh^Sta*^ said that adequate data on project adop- 
tions were not available.) Only 1 of the 47 local school districts 
we visited had. adopted any fc£J>he Network' s title I exemplary 
projects. Most adoptions of exemplary projects are for use in a 
single school, thereby further limiting any impact the Network 
could have on title I projects. 

Local school officials were generally not receptive to adopt- 
ing the Network's exemplary projects, in part because inadequate 
evaluations had allowed the officials to' remain satisfied with 
their title I projects and because they were not convinced that 
the exemplary projects were better than their projects. The ^ 
reluctance of school officials to adopt exemplary prdjects has/* 
been perpetuated by (D the failure of Federal, State, and local . 
agencies to make effective evaluations and (2) .the lack^of in- 
formation about the exemplary projects, which school officials 
consider important to' convince them of the, projects merits. 
Effective evaluations and more comprehensive information about 
exemplary project costs and achievements would help local title I 
officials to more adequately .consider exemplary projects a^ an 
alternative . * ' -> 

However, regarding 'evaluations : 

ED has no^ given States adequate criteria for assessing 

title I project quality • * 

—States have not effectively assessed the quality of^ title I 
projects f and ED has not adequately mOni toyed the States 
activities in this area. 

—Local evaluations of title I project's have bee*? deficient. 

Also, the^ Network' s lack of exemplary"*prbject data' concerning 
student achievement, project costs, impact on student behavior, 
and adoption results has contributed^ to local school ^officials 
lack pf confidence in the projects. ' , ! 
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LACK OF ADEQUATE CRITERIA FOR * 
EVALUATING PROJECT EFFECTIVENESS 

Title I requires that locaL, school districts evaluate the 
effectiveness of their title I Objects. Sections 183(b) and (d) 
of title I require ED to ( 1) develop and publish standards for 
evaluating project effectiveness and (2) prdVide models for 
evaluating title I projects to be used by State education agen- 
cies. The models are to includ^ uniform procedures and criteria 
for local and State education agencies fo use in evaluating 
title I projects. Although this requirement has been in effect 
since 1974, as of ^eptember* 1980, ED had not provided adequate 
criteria for evaluating project effectiveness. 

School districts are required by title I legislation. to use 
the results of evaluations in planniag* for and improving their 
title I projects. Without more specific criteria to assess proy 
effectiveness, local and State education officials* lack a valuable 
tool for determining whether their title I projects need improve- 
ment. One State official said that, without such specific criteria, J 
local officials lack a standard to use in measuring their projects 1 * 
effectiveness. This official believed that local school officials 
would seek better projects (including exemplary 'projects) if their 
existing projects did not meet a specified standard. 

In lieu of more specific criteria for assessing the ef fecive- 
ness of title I projects, ED has issued general instructions £hat 
require local school districts in making evaluations w to 

* * m * 

>*~ US e appropriate and accurate te&ts, 
--use appropriate and accurate analysis procedures, 

— include a representative sample of the- students served, and 

c * • * 

— provide valid measures of title** I student performance com- 
pared to estimates of what the *p erformance might have been ■ 
had title I services not been provided * 

-^'Also, pursuant to requirements' of title I legislation, ED has 

Wescribed uniform evaluation models for school districts to use 
* In assessing their projects. ^ 

Although the above instructions and models provide guidance 
for evafluation procedures, or inputs, they do not provide ade- 
quate criteria for' determining what levels of performance are 
satisfactory. 

/ ED* s evaluation models provide comparisons between the actual 

gains of title I students tfnd estimates of the gains thht the 
students would have realized without title I assistance. Under 
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this 'system, positive scores (gains above zero) indicate that the, 
title I students, performed above what .they would have without com- 
pensatory assistance . Negative scores indicate tliat the students' 
performance was bel6w what* they would have realized without such 
assistance. -ED guidelines, however, do not specify what levels 
Of positive gain might £e considered appropriate, or what levels 
might indicate project lmprov^fentss^re needed. * * 

ED officials told us .that the specification of criteria to___ 
separate satisfactory from unsatisfactory projects is technically 
.unsound and ^educationally indefensible ./ Eft) officials believe a r 
requirement that performance standards be met often results in dis- 
tortions and dishonesty in evaluations and fosters an unnecessary, 
counterproductive adversarial atmosphere between agencies. 

ED NEEDS TO IMPROVE MDNITORrNG ^ 
OF S TATE TITLE"! ACTIVITIES ' 

ED is responsible for assuiryig that State education agencies 
are /meeting their responsibilities under the title I program^ 
Statle agencies have primary responsibility far monitoring ^aG ( 
.evaluating local title I arcrjects., As part of this responsibility 
they are to encourage* schools with deficient title I projects to 
adopt promising educational programs. However, neither ED nor 
States have met theif responsibilities. /Consequently, the^ are / 
not always aware of projects that need improvement ancl, # therefore, 
are not able <to encourage adoptiohs of exemplary projects or to ^ 
provide otl^er guidance for improving title I projects. 

Eft, monitoring of State title 1 ^ * , , " 

monitoring iK limited ~ 

Although is responsible for determining whether States are 
meeting their t^tle 1 responsibilities, it does not direct its ac- 
tivities toward Assuring that States foster effective title I 
projects. ED is \r?quired-by title £ legislation to s 

— Assess whether State applications meet the requirements of 
title I law a&0£ whether the requirements will be carried N 
out. / 

— Submit/enforcement reports to the Congress disclosing the 
(1) ^ctent to- which §tate procedures, satisfy the title I 
Feqtfir.ements, (2) manner ir* which ED's monitoring reports 
wctfre considered in approving State applications f (3) find- 
ings of ED's onsite monitoring visits, and (4) actions take 
/to cprrect^ £roblenis identified during the viaits. > 

/ • ^ *' ■ . 

./ED's administration of title I has not been effective in get?- 

ting States to meet program develppment responsibilities. Histori 

cally, .ED's title I monitoring ha3 focused on whether the funds 



were used properly,, not on whether they were used effectively. 
Consequently, most,' of the weaknesses cited by ED in its reviews 
of State title I activities have pertained to violations of the r 
tecnnical provisions of the^law and not to program effectiveness^. 

ED officials told us that their title I monitoring emphasized 
compliance with appropriate procedures, and not achievement or ^ 
quality. Our review of 14 ED monitoring reports for the five 'States 
we visiter disclosed that virtually all .the problems identified by 
ED monitoring dealt with compliance violations. Ncfne of the 14 
monitoring reports we reviewed contained exceptions relating to 
project accomplishment or disclosed whether project achievement - 
levels -were adequate. States 'also directed their monitoring ef- 
forts toward compliance elements. « • 4 

Irv the past /'insufficient staff levels may have' hampered ED ' s 
ability to adequately monitor title I projects. 'In its 197 & Annual- 
Report, the National advisory Council on, the Education of Dis- 
advantaged Children stated that ED' s title I monitoring had been 
substantially reduced. The National Institute of Educations Com- 
pensatory Education Study 1/ also expressed concern over ED s ■ 
ability to adequately monitor State title I activities. The In-/ 
stitute reported' that ED' s monitoring and enforcement had declined 
-significantly and that States were not effectively supervised. 

ED officials said that*, after .issuance of the Council and 
Institute reports, ED increased its title I monitoring staff fro 
18 to 37 persons. They said that ED intends to put more emphasis 
on monitoring title I project achievement. 

State assessmeiflS-s of project 

effectiveness are not adequate - * 

States are required by law" to assess lodal title I projects 
to assure that (1) project quality is acceptable, (2) local evalua- 
tions are properly conducted, and (3) program deficiencies are cor- 
rected. -State title I assessments, however, generally do, not focus 
on -project quality and are not adequate, to detect ti,tle I, projects 
needing improvement. , , 

State education agencies have primary* responsibility for moni- 
toring local title I projects, In enacting title I, the Congress 
considered States' monitoring essential, to determining the quality 
of title I services at the' local level. States'- responsibilities, 
as>set forth in titles \ and V of ESEA, are to 
,: > 

^."Compensatory Education Study: Administration of Compensatory 
* Education*" National Institute of Education, 1977. 
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— monitor the effectiveness of title I projects, 



, — determine whether local titlesrl projects are of sufficient 
m quality, r , ' 

— -determine whether local school districts are adequately* / 

evaluating the effectiveness of* thfir title I project^, 

i • * * 

A . . . 

— evaluate the effectiveness of title I projects rn improv- 
ing the educational attainment of educationally disadvan- 
tage^ children/ 

* 

*~»assure that deficiencies and noricompliant practices deteg^ed 
through monitoring and evaluation are corrected, and 

--disfseminste and encourage adoption of promising educational 
_y practices. '/ ^ . 

State- agenfcy monitoring activity,' however, generally feocus 
jbn^school districts 1 "cfcmpl^ance" with title I requirements dealing 
witfrsuch matters as allocating funds and selecting children to be 
served. Little State mdnifeoring is directed toward program develop- 
ment or improvement. ' fc 

A 1977 study of title I Administration 1/ funded by the De- 
partment of Health, Education, ahd Welfare reported ,that, although 
States are responsible' for helpina loca'l school districts to im- - 
prove program operations, jnany Stages had not assumed their prqgram 
development responsibility.^ The study stated that: 

— State monitoring of program quality was superficial and 
historically incomplete. * ' 

* 

— States did not have adequate 'systems for title I j^oject " 
development., m ' * -\, * 

— States- did not hold school districts accountable for their 
title I programs and were" noyfc doing an adequate job of get- 
ting school districts to implement effective' title I 
services. 

— State title I evaluation policy was restricted to ensuring - 
that local test data had been submitted. 



. ' — States did ntft question the uses local school districts 
v ' were making of their evaluations and m&de little effort ^ 

to help sch<n>ls link project evaluations tO:jpro>ect, design. 



l/"Jf Study of the {'Administration df the Elementary and Secondary 
Education Act, Title 1/ in Eights States," 1977. ■ \ ' 
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.—States, had not informed school/ districts that evaluations _ 
' . 'are important to project design . ^ 

" —States impJried^ that they did noli* attach much importance- to 
* ■■ program improvement vfren they failed to examine how evalua- 
tions, are used to plan for title I activities. 

State monitoring in all five State education agencies we *V 
visited- focused 6n compliance elements, not project ef f ectiveness\ 
Title I officials, at one State "agency told us that more extensive 
monitoring was needed to enable the State to identify project weak 
ne&ses* and encourage schools to adopt, exemplary projects .' These 
officials -sadd their State lacked title I project evaluation data 
•because it had a small monitoring staff. . Only one- fourth of a 
full-time equivalent-itaf i person was budgeted 'for project monitor- 
ing and evaluation. > » • ^ 

The ^.tle I coordinator in another State said that it is not 
the Stat e mon itors' function' to recommend program changes when • 
weaknesses~are observed. The State' s. function,' according to this 
official, is to monitor for compliance with the title I regula- 
tions^ not to recommend changes or encourage schools to adopt ' / 
exemplary projects. • . 
i / 

Public Law ^5-561, enacted November 1, 1978, clarified - the - 
role of States in title I program management. In developing 
title I legislation, the Congress intended that States consider 
' local project evaluations before approving local applications for 
title I funds. States were expected- to disapprove applications 
for projects that had not raised levels of student achievement.^ 

. local title I officials tend to be reluctant to implement . 
changes to their projects* Stimulus from §tate education offi- 
cials might be necessary £o alert school officials to project 
inadequacies and 'to motivate project improvements. Stamps that 
lack suitable assessments of project quality are not able <£o pro- 
vide such stimulus and are not meeting the" intent of title I leg- 
islation . lx 1 * A 

LOCAL PROJECT EVALUATIONS ARE ^DEFICIENT • "A 

* ' Local school 'districts are required to evaluate the effec- 
tiveness of their title I projects and to. report the evaluation 
results to State" education agencies . Specifically, title I leg- 
islation require^ that- local school districts: 

— Adopt effective procedures for evaluating their title I* ** 
^ projects. 

. — Use objective measures 1 of educational achievement. , 

22 
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—Use the results of the school districts 1 evaluations in 
planning for and ^proving title I project activities, 
* * * * « 

1 . Local school district evaluations varied considerably in na- 

ture a1id~quality. Generally, however, these evaluations included 
various forms of tests to measure student Achievement levels* The. 
local evaluations were usually based 'on standardised horm- 
< referenced tests* In the past, these tests were designed to meas-* 

/Ore the achievement layel of students in 1 relation to the level 
/attained by a nationally representative student sample, The norn^ 
scores were based -on the results of tests given by thd ' t@84maXer 
. to samples of students intended to be^ representative Of the group 
^for whom the tests were designed* As' indicated on page 18, ED^s. 
— new evaluatigm modelk-provide for comparisons between the stu- 
dents' achievement and estimates of what the students would have 
achieved without title I assistance * 

Sehoo^. district ev3±trerfc£^he-r - how e v er^ are seldom used to iden- 
tify needed prpgram impsfevements ( and were often based on method- 
ologies .tttat produce dr^&ccurkte results* School officials' 
limited adoption of tWNetwork' s exemplary projects may have 
stemmed in part fro\n (deficiencies in local school' district^evalua- 
trions of title I activities. Superficial or deficient: project 
* evaluations 'may* fail to disclose ( serious project shortcomings. 
- U naware of these shortcomings, /local officials may see no reason 
—to adopt the Network's exemplary 'project^* Despite* the lack of 
specific criteria tor measuring success, ^implementing ED' s .evalua- 
tion models could improve the quality of. local evaluation models* 

f i 

Local evaluations noj: used -* / - 

for ' pro ject improvement J ^ 

,In designing title I legislation, the Congress intended that 
local evaluations would enable school districts to identify weak- 
nesses in -ttteir title 1 I projects-and would serve as; a tool for 
project revision and improvement • Local title I officials, how- 
ever, tend to resist change to their projects even though some 
project evaluations showed (l) many of the students were continu- 
ing to/fall behind, (2) many needy students were totally 'excluded 
from the projects, 'and (3) many students were dropped from the 
projects ^before reaching the level of achievement of their peers* 
Officials* in 41 of the 47 districts we visited were not interested 
in adopting newJritle T project's* Most were satisf ied^with their 
existing projects and therefore not willing to consider replacing 
' ' them* • 

Studies funded by the Depa*tmer|t of Health, Education , ' and 
' WeJJ^re haye reported that school districts made little use of 
title i evaluations for prefect improvement* Instead, local ^ 
evaluations were used primarily to me'et' title I reporting 

-.' i ■ ■ ... . 
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requirements... The 19 77 study of 'the administration of title I 1/ 
concluded tfiat local school districts — * 

— 'lacked. s^STtematic processes for effectively developing 
title I projects, 

* — gave little attention to project planning aspects of 
their evaluations, and 4 ^ 

—prepared evaluatiorts primarily to meet repprting require- 
ments . 

v Anpther study 2/ dealing with the uses oflocal title I evalua- 
tions reported 'that, rather than critically assessing- their projects 
school officials tended to seek evidence supporting their positive 

feelings and to ignore evidence to the contrary. 

« * 

In appropr methodology redtfces 

the validity of local evaluations , ; _ 

Deficiencies in local assessments* of project effectiveness 
have been compounded by the low quality of title I evaluations. 
We previously discussed this problem, along with the attitude af ^ 
local officials that title I evaluation's ~>re not .useful for assess- 
ing prqject effectiveness, in a September 19 77 report.' 3/ In that- 
report, we pointed out that over half of the State title I offi- . 
cials we surveyed believed that local title I evaluations were 
less, than adequate in terms of credibility of findings*, presen- 
tation of required management information, and qualification and 
quantif icatibn of measurement $ata. Also, our^report said that 
studies by other organizations had shown that: 

# --EvaAfcatiqn designs were not adequate to produce reliable 
data on measurable achievement gains* _ 

— School officials lacked incentives to collect or report 
program output data. — 

— School personnel did not show jnuch ability or interest in 
* . using evaluations to formulate *title I policy or practice. 
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l/lbid, page 27. • 

2/"Local IfaeS* of Title I Evaluations,"' SRI International, 1978. 

2j f " Problem^ and Needed Improvements in Evaluating Offiae of Educa- 
tion Programs" (HRD-76-165, Sept. 8, 1977). 
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Problema were also noted in the way local school districts 
conducted their 'project evaluations during our review at 14 school 
districts. These problems seriously affect the reliability and 
usability of the evaluations to determine whether and how projects 
need to be improved. Some of these problems are the (1) m failure 
toxontrol sample groups in evaluating prQject achievement* (2) 
use of improper methods to compute student achievement, - (3) use 
of inappropriate tests,, (4) failure to measure sustained gain and 
total program effect, 'efnd (5) failure to consider project ef- 
ficiency and cqst effectiveness. ^ * 

t Sample groups not controlled ■ ^ * 

Achievement in title I projects is generally measured^by pre- 
testing students near the start of a school year and posttesting , 
them near the "end. The difference in average achievement level r 
betw'een the students pretested and those posttested is considered 
the average gain for the* subject and* grade level being evaluated." 
For sucff *a measurement tJo'be meaningful, the sample groups tested 
must be carefully controlled to assure that the evaluations measure 
the intended characteristics accurately. 

> - v -* • 7 ' / 

• However, some school districts did nQt control their test 
groups to assiire that students tasted actually received title I 
assistance in the subjlects -being evaluated. As a result, s ome \ 
evaluations measured the gains of students who were not receiving 
title i assistance for the subject evaluated. Many such students 
were scoring, well , abovbs^he national average, on the tests,, thereby 
possibly biasing the. evaluation results. Where this occurs^ the 
evaluations do not provide accurate information on the achievement 
aaia of students actually served and, therefore, are not useful in 
determining whether improvements* aaCe^eeded . r 

J i * ) 

K relate problem was 'the failure to assure that the same stu«^ 
dents were pretested and posttested 4 . Obviously #/ testing one student 
at the beginning of the year and another at the end will not yi^ld 
information pn the gain of either student. However , it was not 
umfeual for as many*as 40 percent of thei students to be either pre- 
tested but not posttested, or vice versa. Some schoolr (Jistricts 
do not consider differences in the makeup of students between pre- 
test and gpstte st groups. Such differences in the te^t groups^ re- 
sulted in "title I project achievement .gains being substantially * 
overstated or understated. ED officials said that guidelines for 
project evaluation now specify that pretesting and posttesting 
must be compared on. ah individual student basis and that, if in- 
structions are followed 'correctly, pretest and posttest groups, 
should be the same. . : _ , 



25 

> 



3& 



m 



Improper methods 1 used to 
cdmpute student achievement 

Some school- systems ,f procedures »for competing student' achieve- 
ment produced distorted measurements. In several school districts, 
for -example, title I evaluators were using methods to compile aver- 
age grade level ^equivalent scores that distort CQmputations of t 
achievement levels. \ p" 

In another district most of the second grade students were 
not pretested, arid ,gain>was measured as if the students had started 
from ^pretest score of .zero. This practice' improperly inflated 
test results because gains achieved over a. 2-yea*r period were re- 
ported as a 1-year gain. ~ 

Inappropriate tests used 

Some schools 1 inappropriate use of achievement tests prpdpced 
misleading or useless- result's*. " In some 'cases, for example, "the — 
posttests anta pretests measured different skills. Some schools 
used tests that were not appropriate for the achievement levels of . 
the students tested. x m - '"N 

So me schools frequently changed the brand or type of test used 
to measure student achievement. At times, the change was made 
within a single evaluation period. Frequent -changes irv typed of- 
tests make it difficult to compare and .evaluate the^ test results. 

\ 

One district had developed its own tests to measure- skills 
taught in the classroom. ' The , title I evaluation system allows for 
the use <f£ locally developed tests where the gain on th^ local test 
,.is translated into" a national metric or scale t However* if the 
V gain on the local test is inaccurate, translations to a national 
metric or.spaJLe will not correct the problem. Analysis pf achieve- 
ment tests by testing consultants showed that most locally devel- 
oped tests were of poor quality. Locally developed tests ar£ not 
thoroughly /tested, as are most of the standardized Achievement 
tests mose^ commonly used for program evaluation. 

Sustained gain anfl total 
program effect not measured 

Although most students are dropped from title I in the early 
grades' and before they have reached the $yerage achievement level 
of their classmates, feft schpol districts have attempted to follow 
tjie students 1 progress to see whether ; ^their title I experience has . 
any sustaining effect. School officials ten<3 to view and manage 
their title I programs as annual efforts, rather than as a series 
of integrated units s designed * for cumulative impact on the stu4ents 1 
V academic success. None of the districts fxi our review h^d analyzed 
the cumulative effect of consecutive years of title I assistance 
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on the rate or student achievement gain. Similarly, the districtt 
had* not attempted, to follow students after they Vere released fror 
title I to^see whether\the projects had any lasting effect on stui 
dent achievement, dropout rates, attitudes toward school, or othei 
factors. 

i • 

" Some impetus has been given o £o the measurement of sustained 
gain by the Education Amendment s°'of 1978. ESEA title I noy re- , 
quires local project evaluations to ^Include measurements of 
achievement over at.- least a 12-month period at least once ^uring 
each 3-year period. School districts are required to use /this , 
information by considering the inclusion of project ccmpohents . 
designed to sustain student achievement beyond the school^' year in 
which the projects are conducted. 

Program efficiency and cost 

effectiveness not considered; > v 

• Only one school district we visited had attempted 7 to analyze 
the cost effectiveness of the various title I projects offered in 
its schools. Education officials testified that furiding limita- 
tions ^have caused school districts to exclude many eligible stu- 
dents or germinate their title I services. Measures of cost . 
effectiveness' could help school officials determine whether their ■ t 
projects are efficient in comparison with other schools- and dis- 
tricts or with exemplary projects. 

•f . 
Agency actions, to improve evaluations 

ED has focused substantial effort on improving title I 
evaluations' .' . In compliance with a requirement .originally added 

' to ESEA in 1974,* section 151 of Public Law 33-380, ED developed 
evaluation mode^and standards for use by State and local educa- 
tional agencies** \ On Octobers 12, 1979, final regulations were pub- 
lished specifying 1 models for evaluating the effectiveness of title I 

' projects providing instructional services in reading, language arts, 
or mathematics. ED's three evaluation models were developed to 
provide reliable da>a on program accomplishments. The models were 
also designed to-permit the aggregation and comparison of project 
results, even though the school £istricts^use' different kinds of v 
tests . * 

In 1976, -10 regional Technical Assistance Centers were estab- 
lished under contract to ED to provide' evaluation technical assist- 
ance when* requested by States and their local school districts. 
The Centers werj established to assist States and, at the dlscre- 
.tjjbh of the States, local districts in implementing the title I 
Evaluation and Reporting System and in dealing with other tit lft .1 
matters. The Csntters' function is to improve / through training*- 
and consultatiori , State and local, capability for performing title I 
evaluation, technical, assistance focuses attention on the evalua- 
tion models andjreporting forms included in the title I evaluation 

% - / 
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* system. Training consists primarily of evaluation workshops 

* requested by the States, intermediate service .units, or school 
districts • Consultation is provided to facilitate the actual 
implementation of orter-dr more of the' title I evaluation system 
components in site specific situations* The Centers also provide 
assistance in dalfe utilization and quality control, ,suq.h as adher- 
ence to testing dates, accuracy in completing forms, score conver- 
sions, data analysis, and data aggregation* 




NETWORK DATA ON EXBfPLARV • 
PROJECTS NEED 'IMPROVEMENT ^ . \ 

Readily available information on Network exemplary projects . / 
often dpe^ not show the projects 1 effectiveness based on (1) the '%/ 
developers 1 experience or (2) the. experiences/of schools that have y 
adopted the projects. The* absence of convincing data on the -merits ' 
of 'exemplary projects has contributed to school ^officials' opposi- 
tion to /adopting exemplary projects. 

/ * * * 

Moreover, the impact of , the Network on title. I projects may' . 

be Overstated because the Network's accounting for the number of , ' ^ 

adoptions for title J projects included adoptions that did nqt tafte 

plac^ ahd adoptions for other than title I projects.. , 

Dat£ oft the merits of Qjcemplary 

projects inadequate + — - — — — *- 

7 A brief summary description of each'Network project is in- 
cluded in a catalog entitled^ "Educational* Programs That Work i 11 
The ^catalog, issued annually' by th^ Network, includes the name and 
telephone number of the y project director, Vho can be contacted for 
further information, tiowever;, ED Officials stated thatM^heir ex- 
perience, with the Network was that few decisions to adopt^a Network 
project were ma^de as a result of reading printed materials. Ac- 
cording to ED,^a study found that most decisions to 'adopt an exem- 
plary project -are raade^as a result of representatives from a school 
district either visifefng a project to see it in operation or at- 
tending an aWarenesI* conference and talking with the project 1 
developer . ' ' % „ 

\ # / > , ( 1 

Most school officials we interviewed believed that the Net- % ' 
work 1 s exemplary projects were not superior ito their school 1 s # 
ti'tle I projects. Available Network literature about the exemplary 
projects frequently lacked -convincing da.ta to establish their super- 
iority. Without' such data it is questionable whether school offi-/ 
/ cials who were not already interested in a project would visit the 
project or attend an awareness conference coficerniiig it. Although 
project developers provide information demonstrating their projects 1 
effectiveness to # the Joint Dissemination Review Panel, that informa- 
/bion is not usually made available to prospective project adopters. 
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. ' .. . . .... . 

;pur review of the cataldgs and .other Network literature .showed th^t--- *•> 

•• , data on sustained. achievement\galn were available, for onj - 

' 1 of the 21 exemplary title I projects, . . ' 

• . * , - • 

• , ' —literature. for 6 projects was devoid of any cost- 

L " ef fectiveness data, and , 

, —information on student behavior attendance was/npt 

"T 7 "" * available for 18 of the 21 projects-.., j, ; J 

• " ' • Lo-pal title-I officials, at" 34 s'dhWUistricts^'ia us that. - 

one reason they had not adopted exemplary-^le^fro ject* .was • •■ 
that they could not evaluate whether the projects were suitable 
1 * fo? their districts. Officials oC .13 6f the districts told us 

£hat the literature distributed: by. the Network and the developer- 
demonstrators on exemplary projects was not convincing . For ex 
'ample, officials of one. district s^id they considered-adopting, an - 

• ' exemplary title I project buV^decided against it because the 

!Sted^riformation available about project results was 
cient to ^onVince them that the proDect would 'be better than what 
• ~ they had.* .DistSct officials hoped to see a record of the exem- 
, plary projects' effectiveness over a period bf years. 

. ? * Even some of the school districts that adopted exemplary 
- * - „ projects complained about the adequacy of information ^^ning 

the proiects. - O f ficials at ^eraJ of these di stric ts told us that, 
Sfo?maiISn ihey desired to see on the long-term effects of the ex- 
emplary projects on student learning performance was not available., 
SnVof filial stated that the limited information that was available 
before a project was adopted had made it difficult to convince . 
school staff and officials to 'adopt the project. ^ ' 

Officials of another school 'that adored *n exemplary Project 
said detailed information on projects had been difficult to obtain. 
They said that the information they were able to get on P^ject 
effect, sustained growth and attitude, attendance, and behavior . ^ 
V ' improvement was mostly unsubstantiated, opinion. . + 

v*. Also, some State "facilitators told us that, because of tfcfe ' 
lack of .project. data, they were not able to respond , to school ofr 
' ficials' questions about exemplary projects. Various State fa- 
cilitator officials-expressed concerns oyer the lack of 

- ... \ 
—data to determine which projects were best suited to 
particular school districts,^ 

' —data on the cost of operating some projects, , 

--data that would enable school offici-als tb make project 
comparisons, . * . •, 
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— data pn prpject implementation requirements that would 
enable of ficials 9 to 'assess their schools 1 abilities to 
, implement' the projects, and 5 ^ 

,-^curfent data op projects . 



Regarding the currency^of, data, our review v of project li^qra* 
ture that was 'dated disclosed that about one-fo„urth of the projects 
were presenting data 4 or more years old. In Sine case, student 
acHieyement data wtere 11 years old, ED officials told us that they 
generally did- not obtain performance data fox the exemplary pref- 
ects after the year the projects were certified by the Joint Dis- 
sem^ninritfn* Re vieW" Panel, Nineteen of the title I exemplary proj- 
ects .had bden presented to the panel before 1975, Consequently , 
currenttdajta had not* been available to ED or potential adopters 
to demons tr a t«^t.hat the? current project formats were producing the 
same exemplary > results • 

* ■ * € * » » • • * * 
Feedback, on effectiveness of * V/ 

adopted p&ojectgt not obtained * 

Hie Network has not routinely obtained ' information ^8gthe re- 
sults achieved by'the districts or scj|0£>ls that/ adopted ^pftp la ry 
projects, - In our opinion, such information would be useful to the 
a Network fn promoting adoptions q£ exemn&ary jpro jects by demonstrat- 
ing that ^exemplary achievements can beF replicated at adoption sites. 
* ■* 

# * Although a 'ma jor purpose of the Network -is. to encourage im- 
prpvera^t^jUirough the adoption pf exemplary projects, virtually 
the only ajjfffcfrmat ion tte Network^ had on past adoptions was a list 
of them. StSTxhe time ifb our fieldw&rk s , She* Network did not have 
, a formal .system to acquire information on the success or failure 
♦of ;title I project adoptions. 

* the Network director told us that the Network 1 s efforts fo- 
cused 6n getting schools to* adcypt exemplary projects. Fdllowup ac- 

* tivfties were not considered #a high priority, and little followup 
had been done. Officials of five State facilitators and six 
developer* demonstrators we interviewed indicated that they gen-' 

• ef^lly ,diS not- follow up to see what happened af ter^ a district or 
school , agreed t'o adopt # an exemplary project. 

* Our* interviews with the five State facilitators disclosed 
that - * a 




— none could provide information on how adopjted exemplary 

projects had affected student learning and achievement 
. levels ; ♦ 

--four did not know whether some of the reported adoptions' 
had been implemented; 
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— three believed a better, followup system was needed^ to 

evaluate the results ,of past- adaptions, but said that f61- . 
lowup efforts on -adoptions would require' greater funding * 
for additional staff; % 

— one said that with additional funds he^could^m&ke ev^luS^^ 
tions, of' toroject- quality,' which are not now provided in the 
contract* and 1 - / 

—one .said ttojt, because his office has only two' professionals, 
not much foliowup can be provided . # 

Most, of'thei seven developer- demonstrators we interviewed also, 
cited a need % for better followup to evaluate the effects of adop- 
tions and to assist 'in adoptions. However, only one. developer- 
demonstrator had followed up to Obtain data on the accomplishments 
of adopted projects. The Network published this information to ; 
publicize the impact of the adoptions. Officials of the other * 
five developer- demonstrators indicated a. serious ladk of informa- f 
tion about the results of prior adoptions. In, this regard: 

— KDf f icials^ of four developer- demonstrators said they performed 
little ofc^no followup and could not tell us how many adop-* 
* * tions of their projects -were, still in effect. Three of them 
• were not sure whether a^.1 of their claimed adoptions had 
^ been implemented. In^contaeting 15 of the schc^Ql districts - 
that a developer- demonstrator reported had adopted projects, 
we found that 4 had never adopted the project and 2 had 
dropped the project. Officials of- the developer*- demonstrator 
^were ndt aware of these six case^ or the reasons the projects 
were not operational . * 

Art official, of another developer*- demonstrator told us that 
it asked* all School districts adopting its project to provide 
student achievement data £or use in, evaluating the success 
of adoptions. However, only „of over 140 adopting districts 
submitted the requested (&ta. *The official stated that the 
•v developer^- demonstrator depends- on the adopting schools to 
\ submit information voluntarily, since it has neither the - 
' time nor the staff to follow up. * Unlees>the adopting schools 
contact it, the deVeloper- demonstrator does not kno,w if th6 
project wafc^xlropped or even adopted. , 



"Became of "the importance of f eedback fc data ori project adop- ^ 
tions, jgome deveroper^-demohstrators said the Network should provide 
greatei^mphasi& $nd increased funding ^or periodic followup $nd* 
for the^val^jat^on of* adoption results. • * \ \- . * - 



^ve^ral school district officials stated that the -Networ)^ 
lack of assistance' for followup. activitie^has adversely affected 
their adored projects. According to these officials, the 
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network 1 s trairiing and f ollowup efforts were inadequate when pro- 
ceeding through critical siages of the project. An official of 
one school district said that, if the Network wants to keep^the 
adopted projects from being diluted and keep the "momentum going," 
the personnel of school districts adopting exemplary projects 
should be retrained periodically* - Of f icials, of two other school 
districts that had operated exemplary projects for 2 y^ars without 
any support from the developers said theywould like postadoption 
assistance, including access to material, techniques, and feedback 

fropi the developer. 

* • 

The »eed for ' careful , thorough implementation of exemplary 
projects was emphasized by a 4-year ED- funded study of Federal pro- 
grams supporting educational change,. 1/ Although some successful 
replications were observed the study concluded that 7 

— successful projects were not disseminated easily, 

-^replication at new sites usually fell short of performance 
at the original sites, 

^--few projects were successfully implemented, * and 

— fewer survived fn the long run. 

Based on an analysis of the factors influencing the success 
of the adoption processes, the study reported 

- — implementing strategies can* make the difference between 
success or failure, 

» * ^ 

— implementing strategies can determine whether teachers 
would assimilate and continue using project methods or 
allow thfcm to .fall into disuse, 

— one-shot preimplementation training^ is 'ineffective, # and 

—extended training* a$d classroom assistance from project 
* staff are effective implementation strategies; 

Uxe study concluded that the inability of many dyctoool districts 
to. Implement and sustain program changer ultimately frustrates, the, 
objectives of Fe'derarl education programs. Instead of cbnaentrating 
only on the initial stage of^the adoption process, the "study reccrni- # 
mended that the Federal role be expanded. to subsequent adoption j 
stages ifi order to assure the success an<? long-term institution-; 
alization of adopted projects^ ' j 



< 
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p^kting Educational' Change, Vol. VIII: 1 



l/'^Federal Programs Supporting Educational' Change, Vol. VIII: 
* Implementing and Sustaining Innovations," RAND Corporation, 
• 1978. - < 
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Better information needed to assess 

the Network r s Impact on title i" * ^ * ? 



Accurate information on the number of schools adopting Network 
title r projects would .provide a jnore reasonable basis for deter- 
mining (1) "the extent to w"hich the Network's projects are being 
used and (2 ) the need for any adjustments to .increase the Network 1 s 
potential for improving title I projects. t 

From its inception in 1974 through the fall of 1978, the Net- 
work approved and funded for dissemination 21 exemplary title I , ^ 
projects. . The Network, howfever,~has not determined how many, dis- 
tricts and schools have adopted the 21 projects for tifcie I 
projects. The Network received ^reports on adoptions X>r exemplary 
projects frojn its developer- demonstrators, but these reports did 
not provide an accurate basis for recording title I adoptions, 

Developer- demons tratorsfejfrequently reported only the number 
of districts that adopted projects* without indicating the number 
of schools adopting in each district; therefore, the Network did 
not know how. many school^ adopted j^pro j«ct in each' reported adapt- 
ing ^district . According tq>Elf, memt adoptions of exemplary projects 
are'not for use -in .schools throughout the district and often are 
ortly for use in a single school/ Reporting on a district basis* 
tends to make the Network* s x^orted accomplishments appear more 
extensive tjian actually is tfhe case. 

Also, developed demonstrator reports did ,not prgjp.de assur- 
ance that reported adoptiqps 'actually occurred and were continued. 
Our followup on 36 aQoptiohs recorded^ by State officials showed 
that 5 had not occurred* and 4 had been discontinued* The Network 
did not adequately follow up to verify that districts and s schools,, 
had adopte^and^^continued to use exemplary projects as reported, • 

Moreo^r, as previously stated, reported adcfctions of t£tle I 
exemplary projects include adoptions for use in other than title I 
activities. Our followup of 87 adoptions that occurred in four 
States we visited showed that at' least S9 were not for use in^ 
titl$ I objects, .While we do not question the desirability of 
using title I fxemp$ary(projects; for^other than title I projects, ^ 
we believe that data orfxhe Network* s impact ty6 improving title I 
projects should distinguish such adoptions from those that replace 
or modify exist±pg~J;itle' I projects. * ^ c 



These x^jK>rtin^ errors resulted in the accumulation of u-n- , 
relX^ limited its usefulness. 

Network actions ; 
*• 

,. To improve developer-demonstrators* foilowup on project ad6p- 
tions, the ^Network in April 1980 fevised its operating regulations 
to specify* that grant applications for developer- demonstrators 
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will be examined to assess ^thfe extent that plans, "show promise of 
effective postadoption moniroring and evaluation of program -impre- 
meritation and resulting benefits at the adoption sites . l \ 

The Network regulations, however, did not specify the nature^ 
of .the monitoring and evaluation to be performed ox the data to be 
-collected during the postadoption visits* Network officials also 
pointCTt out that the current funding level of each developer- 
demonsxrator is not \u^icient to support all activities for wh^ch 
it ,cotfld be responsible.. , ♦» ■ * f / ''^^a* 



* '1 The Revised Network regulations also providd^that the Joint 4 . 
Dissemination Review Panel 1 s . approval of exemplary^ projects expires 
at the end of 4 years. Projects applying for theJP^nel's reap- /, . * 
proval now must"' provide evidence that! Adoptions have attained .the,, 
outcomes originally stated by-^the project developer . - These require- 
ments should make exemplary project data mor#4current and provide 
dat*a on the results of some project adoptions. . The Network instruct 
tions, however , %N do not specify the nature or type of data *to be . 
submitted.' * ' . 

Revised instructions for fiscal yeai?— £981 applications for 
Network- developer^- demonstrators specify that applicants seekih^ 
rerf£wal of their grants must provide data on' the results of. their 
operations, including m - 4 

— the number of students in direct contact .wath teachers "* 
that received services from the" Network* developer- 
demonstrators, . 

—the number of schools provided services by the developer-' 
demonstrators, and , *" - " ; " 



« 

—an indication as to whether the adoptions have produced 
/a significant impact. 



4 Ttiese data will give the Network some measures of the 
developer- demonstrators' performance during the operating year.* 
flbWever, the reporting format does not indicate whether the adop- 
tions are for title I projects,. Consequently, the Network 1 s 
impact on title I projects will remain unknown. 

* 

Whet^hfer the Network 1 s revised regulations and instructions 
imgjrove developer^ demonstrator followup activity and produce' data 
on* the number and .effectiveness of adoptions will depend on the 

>^xtent to which the instructions are carried out. The Network's 
instructidns do^ npt xietail how the followup .and reporting should > 
be accomplished^. * Also, it is not clear how the developers are to 

s report the status of adoptions implemented in prior years. 

-~ r 

/ - 
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, • CHAPTER' "4 * 

/. , • ' ' COifCLUS 1 0N$',' RECOMMENDATIONS ,- 

< ; * *' ani> Agency REspbNSE 

^CONCLUSIONS . . 4 \ ' *N J"^ v J 

-AlthovijCh many, title I te proj4ct^are £mproying.the^acad'emicr 
achievemints of educationally disa|pantage<i students, ^some proj- 
ects are npt successful, -"and services for Sost students are ter- 
minated wjiile- thdy Are still far below the. academic achievement of 
their ppn-title I /Classmates* " Soma school districts have adopted 
exemplary 'project for title I projects, but the- n.umber is tpo 
small to have mjach impacft on the ^ef fectiveness, of title I' in meet- 
ing its goals yi^tion^ide^* / \\ ^ y 

Ina£eqtaacies in title I evaluations at the Federal, State, 
andxlocai/levels are contributing causes* J6r the relatively few ' 
adoptiprfs for title I projects. ED' s title I evaluation and re- 
porting 6ys tern provides the methodology for evaluating the title I 
projects. ..However; ED has neither prescribed adequate criteria 
for State and local officials to use. in determining the adequacy^ 
of their projects nor effectively ;monitojtred* the States 1 title I * 
administration. The 'State education agencies", in tuirn, have pro- 
vided* only limited monitoring of local .project results*. Locaf 
title I of f icials, have seldom' used "pjsoject evaluations as a tooL 
£q£ project planning and improvement. Sometimes inadequate local 
evaluation methodology has limited the 'reliability and *us ability 
of evaluations for identifying project weaknesses. Unden such 
circumstances, relatively few local : school districts have sought 
to adopt exemplary, title I projects . m 

ED's actipns to implement the new evaluation models should 
help to improve the reliability and; comparability of title I , , 
evaluations. However, project inproveraefit depends on local schoof 
officials' receptivity t to change. Given local school districi^ 
officials^ satisfaction with their tj.tle I projects {see p. 23) 
and their tendency to seek information that supports their posi- 
tive views and ignore information that £oes not, few school dis- 
tricts are iikely to adopt exemplary pfcp^ects. Consequently, 
unless the States strongly emphasize using title I evaluationa^for 
project improvements," the Network is unlikely to have much impact- 
on improving title I projects. 

A paucity of information about tjie capabilities of exemplary 
projects an# tfe$ results^ achieved by schools adopting such proj- 
ects also has led iocal s*chool officials to believe these projects 
are no better than. their own.^ School officials wou^d be more 
likely to adopt exenplary projects ,if more <:omp3^eHensiiv4 informa- 
tion were provided about the merits of j3uch preset p a*\<| the re- 
sults achieved by schools that adopt the projects. 




Because implementing educational practices is a complex, 
difficult process, ^we believe systematic followup ,of adoptions 
is needed to determine whether adopting school districts have^ 
successfully implemented the projects. In the past, a serious 
shortcoming has been the Network r s lack ofi^ a systematic followup 
.program to ensure that (1> projects are pnoperly installed and ■ 
operating in the manner the developer believes is necessary for 
success and (2) additional assistance is provided when t/he* projects 
are faltering. Revised Network (instructions emphasize the need 
for monitoring and evaluation at the adopting si^s and providing 
.data on project outcomes.. The Anstructions, however, do not 
specify the nature of the e^a^atipn; an<k monitoring to be performed 
or the data to be provided. i£ccers(iftfj4K', the instructions 1 value 
will depend on how aggressively tne Network pursues compliance X** 

A more Extensive followup progVam at schools, adopting Network . 
projects would v improve' the accuracy pf the data on the numbe* of , 
adoptions. Improved data -would help .the Network assess the, inter- 
est in its projects and the need for improvements in its operations, 

RECOMMENDATIONS ^ . * 

To improve" the procedures for identifying and correcting 
title I project weaknesses, the Secretary of Education should 

-'-provide substantially increased -guidance and technical 
assistance to State and local school officials i t n develop-/ 
ing criteria for assessing the effectiveness* of title I / 
'projects; ' ' , 

— expand ED's monitoring efforts for assessing whether 

State education agencies" are (1)^ meeting their repponsir 
. bilities to evaluate the effectiveness of local "title I 
* projects and (2) helping local school districts to improve 

. * their title I prpjects; \j 

» ~ \ 

— direct State education agencies to (1) assess the validity 
of procedures use<£ by local school districts to evaluate 
, their, title I projects, (2) prespribe corrective measures 
where evaluation procedures are inadequate, and (3) help 
lo£al school officials use evaluations for detecting and 
correcting projeqfc weaknesses; £nd a# * 

—advise State education agencies to encourage Local school 

offi cials €o use iretwork assist^jace in improving, ineffective 
> and inefficient title I projects. 




T<r> convdwfi^pp*rer>tial adopters of the merits of exemplary 
projects, the Secretary of Education should 



--direct the Network to provide qpmplete, comparable, and 

'current information about the capabilities of the Network's 
- exenq?lary, projects, Including (1) their impact on scholas- 
•< tic" achievemertt", student behavior, and sustained growth, 
(2) their costs, and (3) the results of past adoptions by 
other schools. 

To provide more accurate data on* the volume of title I adop- 
tions of exemplary projects arjd ensure that adoptions are success- 
ful, the Secretary of Education should make sure that t the N^fctfork 1 
developer-demons t r ators * >, 

» * * * 

— provide accurate data on the number of adoptions arid the 
'schools affected, . • 

f — determine whether the pro^f&bs" have been installed cor- 
rectly and are functioning properly, and 

—provide for additional implementation assistance where 
needed. *~ 

AGENCY RESPONSE AND OUR EVALUATION 

In a draft of this report, we proposed t*hat ED prescribe 
specific criteria for State. and local officials to use in assess- 
ing the effectiveness of title I projects. In it£ April 21, 
1981, respbnse 'to our draft-^(see app/lll), ED disagreed with 
our proposal. 

' ED said w therSiis no legal requirement to* provide criteria 
for determining what levels of performance are •Satisfactory . 
Section 183(f) provides^ that, ED title I evaluation models must 
speciify objective criteria for use in evaluating title I programs 
and outline techniques^ for producing data which are comparable on 
a statewide and nationwide basis. ED contends that, under this 
prc>vision, it -is* required to provide criteria or standards to 
help school districts choose methods by which they can evaluate 
their title I projects. ED believes that its' October 12, 1979, 
title I evaluation regulations provide models and technical stand 
ards for evaluating title I projects and that this satisfies the 
statutory requirement to provide evaluation criteria. 

However, we believe that ED's obligation in fulfilling its 
requirements does not end w^thythe' publication ^of Revaluation' 
models arrd technical standards) ED should place increased em- 
phasis on providing technical assistance at the State and local 
levels in developing criteria for assessing the effectiveness of 
title I projects. This would be consistent with ED 1 s longstand- 
ing policy of nonintervention in the programmatic decisionmaking 
process at the State and local levels. * . 
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According to ED, implementation of th^ title I evaluation 
system will permit loca* school districts to assn&ss their ti£le I 
achievement gairis on the same metric or sSale arid thef efc^re "permit 
States to review results across school* districts . ED sai<^ that 
national aggregation* will enable Stat^'and local education agen- 
cies t£ view the levels of their gains in light of the ranges of 
g£ins reported nationally. "Additionally, we believe that EJD 
should use^the national, regional, and district level informa- 
tion that it collects to-provide increased guidance to State £nd 
local officials' concerning criteria for assessing the ef fective* 
nessof title I projects-, and we have revised our recommendatioi? 
accordingly. * 

^EH agreed with th§ other recommendations in this report, - 
'Its comments are discussed below. 

Expand monitoring effqrts for - % 

assessing State evaluations and { " s 

assistance to school districts — 

: ? " — * — : ■ = 

ED agreed with our ^recommendations and said that it places a 
high value on the States 0 responsibility to evaluate the effective- 
ness of local /title I projects and to- provide assistance to local 
school districts to improve theii; title I projects, 

- ED pointed out 'that! during fiscal year 1980^ its reviews of 
State education- agency activities devoted greater attention than * 
in past 'year's to title I\Waluation and that it intends to con- 
tinue this ,4ffort to enst^ps that States and local education agen- 
cies meet title I evaluation requirements as well as to improve 

the quality and qse of evaluations at State and local levels. , 

/ 

.According to ED, it reviews annually the. States/ administra- 
tion of title I, assessing how well they are meeting' their evalua- 
tion responsibilities and recommending actions to improve their 
overal/L\ administration^ of ^bitle I. * ; 

Direct, State education agencies \ ^ 

to reprove the validity 'of ^ 
evaluations and assure their use 

incimproving title I projects J I £ 

7 

1 ED concurred and suggested that the thrust of oqr recom- 
mendation was { being met through ' 

* ✓ ' '* t 

—ED reviews of State administration oi title I, which 
* - include recommendations, as needed, /for^improving 
evaluations; 
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* -^mplementaticte and use of ED's prescribed evaluation . 
Vodels 1 and standards; . and • 

--the evaluation services* available* to State and local edu- 
cfation agencies through the regional Technical Assistance 
Centers. , • " - „ ** 

These efforts should help improve the validity of local eval- 
uations_and make them more usable in identifying and correcting 
project weaknesses. However: 

— Reports on ED reviews in the States we visited seldom indi- 
cated that evaluation procedures had been reviewed, and 
none "of the reports we reviewed made recommendations for 
correcting inadequate *or improper evaluation procedures. 

\ — Evaluation procedures _ were technically unsound % ln many dis- 
tricts we visitfd (including districts- which had profes- 
\ I sional evaluators and had implemented' the required evalua- 
_ tion models ) ; j * 

— The Technical Assistance Centers do not have, the" authority 
or the capability to monitor I and correct local evaluation 
efforts. Th^fey can advise St&te and locals education agencies 
only when h£%p is requested. J Tfafy, cannot conduct the evalu- 
ations, perfqfcm the analyses!, or ^interpret the evaluation . 
- . -^results for the St&fces. The! Centers operate in a support- 
ing role, not one of supplanting State, x and- district respon- 
sibilities for completing tl>e±r annual evaluations. 

* — Technical Assistance Center ^reports suggest that the Centers 

do not have the capacity to .provide ^JLl the assistance rieeded 
/ by the local school a districts. For Example, one Center re- 

ported that it had provided 1 assistance to less than 32 per^- 
cent of the districts in it^ area as oi September 30, 1979. 
Another Center reported that States should be encouraged to S 
* h«ve a backup system for training local school district 
officials in evaluation to .'supplement the Center. 1 & workshops. 

— State monitoring of local title J projects has been; compli- 
ance oriented >nd has been j ineffective in assuring the 
validity of loc^ evaluations. . Thia problem was, strongly 
suggested, by tne wides£rea<3 use of inappropriate and faiilty 
* evaluation procedures in t^e districts we visited. . 

Accordingly, we believe^ that «ED needs to strengthen -its 
. efforts to ensure that Staffe education ageusies are properly 
carrying out their evaluation responsibilities 2 . 

*• ,/• * /• t / 
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^Encourage the use of the a 
Network for title I projects 

0 

ED agreed with our recommendation and stated that exemplary 
title I program? disseminated by the Network should be ^Included 
in each State % Bj repertoire of strategies for improving teaching 
and learning in title I schools. ED baid that chief "s^ate school 
officers^ Sta^/e title I coordinators, land State facilitators 
should encourage title T districts to \ examine exemplary -title I 
programs diffused by the Network for .possible solutions to their 
school iraproyeraent problems. According to ED, this encouragement^ 
should gi^ye /school district representatives opjpqfrtunities to meet/ 
face-to-face with the developers of many exemplary programs. Thifs 
process/ according to ED, gives district representatives oppgr-' 
tixnities to examine materials and ask questions about eyaft^Kio)! 
results, installation costs, program philosophy, and training 
requirements. - - V J i /* ' I * 

ED-plans to, continue its ef for/ts^tx^^Acourage State education 
agencies to workjwith local education ag^nAes to, assees the effec- 
tiveness of local title I programs, in^rove^iocsl projects u ------- 

the adoption of exenqplary projects^ encpurage local educate 
agencies to critique their own resylts, and consider the ' 1 
of v exenq?lary projects. ** v ^ 

ED believes encouraging school offi9ials who are 
fied with theirlprograms to attend awareness conferenc 
crease the rate of title I adoptions. ED will contini^ 
with State education agencies to stimulate interest on, the/ part 
local edudationfagencies to .analyze results of theirv^urrent pro;; 
^cts and- -to consider adopting exemplify projects, if appropriate 7 

, * Provide completje, comparable, and ■ *- / / 

current inform ation on Network projects 

ED agreed Jwith our recommendation and stated t4at,/ as Required 
by Network regulations, by 1984 &qr£ eadh pifojec/t t$ retain/ it^ ( , 
exemplary status, project da,ta must be submitted to the Joint Dis-p 
semination Review Panel documenting that the yresultsj 4ro/stirL putt- 
standing, that] there have been adoptions ofyfche Jjkoj&Yt ana that: 
the results achieved at thfe adopting sites are also outstanding. 

ED said that the use of title 'i evaij/atioq i raocUsds by/all I 
local education agencies will result in Panel §ubii^ssion^\wi^ v 
uniform coitparable evidence of achievement intact. / ED Vddeji ;th^at * 
the Panel has] established a format for /submission lot bath news ; \ 
^projects ^nd those being revalidated tfnich contain^ a specific \»- 
format £or\re£orting data on pro jectystartup ^nd /o^er;ating^ costs , 
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sustaining achievement giiins 
dered in' reviewing the ,Impact 



• ( ED/ moisted/ out that, although 

v <pve/t time le one aspect to be consic _ * 

<?f /anyexeWlarj/ project, <fche methodological problems in cpnduct- 
iflg a/ lc^gdtudinal study are considerable, however, ED believes 
tnat/some^ evidence of sustained effects will be available from 
each exemplary/ project as part of its resubmission to the Panel* 

/ ED /claimed that potential* adopters rsfise few questions abou£ 
project- evaLu4ti on results and doubted whether the publication j»£, 
- 7 jte arid costly information^ j about progz/ams^ will result 
feases* iji the number'. Of adoptions. However, ED bdri- 
»very exteraplary project should make information avail- 
ing, project results and agreed th^t the one-rpagc 
^ri*the "Educational Programs That Work" catalpg can 
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.be amp/oved7to"''prbvide more information. , 

/ / / * 
Strengthen /followup of 

exemp lary project adoptions 

ED .agreed with our recommendations, ED said it plans td-raake 
/ greater use of the developer-demonstrators to monitor claimed 
loptions in each State, to gatherj information on adoptions ' 
ncluding. achievement results), a;nd to maintain contact with ^ ■ 
^iopting £ites . ED added, however, that* the current funding^ level , 
of elch developers-demonstrator is ;not sufficient to support all o£' 
the activities for which a developer-demonstrator could be respon- 
sible > in all 50 States. The Network is attempting to provide, this 
support witriin its funding 1 imitations f ED said it is placing 
much more emphasis on the 'quality! of the imp.lementatiort and the 
persistence- of installations than' was clone in the Network's early 
years. ,Also, the number of developer-demonstrators funded by the 
Network is being reduced so that the average level, of funding can 
be increased slightly to make additional resources available for 
monitoring and followup implementation; assistance where needed. 
ED plans to require more* information in the future from each ' 
developer-demonstrator /concerning the status of adoptions in ^ 
order to strengthen and maintain 'the linkage between developer- 
* demonstrator s < and school districts that adopt projects. 

According- to ED, it recogniizes the 'value of involving the 
developer-demonstrator as, an active participant in, the adoption . 
process and followup, and the d£ve|oper-demonstrators currently % 
maintain contact with their adopting sites 'by telephone s and Bite 
visits." ED stated that, when 'resubmitting for Joint Dissemination 
Review* Panel approval before 1^84, each developer-demonstrator will 
havl to include achievement data from five adopting. sites reflect- 
ing the diversity of sites whi^h have adopted the project*' • 

ED stated that the Network collects accurale data on the 1 
number of' adoptions, the jriumber of schools, and the number of 
teachers and students ^ participating in the adoptions during the 
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/reporting period but does not collect data oh which adoptions 
/are title I [projects or the^ number of title I schools a.ffected. 
> ISD explained *that # when the present adoption reporting forms were 
4 cleared', pressures to reduce paperwork and^ reporting Tequireraents 
prevented the Network f rom- collecting information on the numSiws* 
sources of funds, including title I funds, used to install Nk 
projects, ED intends to. request clearance from the Officg of Man- 
agement and Budget to collect information on the particular fund- 
ing source used to adopt an exemplary project. 
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APPENDIX I ~ ' *• -APPENDIX- 1- 

* .* - " 

^ELECTION OF SCHOOL DISTRICTS . 

v FOR SeITE VISITS 
^ it r 

selected a judgmental sample o"f 47 local school 
in five States, for sitfe visits. These districts were 
ascertain (1) whether local school districts have 
for assessing the need for title I program improve* 
action districts take to adopt better title I .projects, (3) x/ what 




imp$ti£ the Network has on improving the- districts 1 programs, and 
(4^^hat improvements district officials believe are needed in 7 



disseminating inforpiation oh exemplary projects. * 

The *dia€^icts selectedAncluded a wide variety, ranging from ' 
very small to very large and from very rural 'to highly urbanized. 
'Care was taken to ensure that the number of school districts re- 
porting a^ove average distriptwide afchievement scores approximated 
the niopber oi districts reporting, below average scores , Because 
title I was created to assist school districts affected by high 
concentrations of poverty, we did not select the more affluent 
sdhool districts for site visits. For this determination,* school 
districts having smaller than aVerafce proportions of^ low-income 
families for their respective States were not selected. > 

The sample of 47 school districts, which had a combined en- 
rollment of about 1,400,000 students, included. 

— 6 large urban districts having ^s^llments over 50,000, 

" — 17 medium-sized districts having enrollments between 
8,000 and 50,000, and / . * 

— 24 small districts having enrollments between 146 and - 
8,000. ' ? % 

The seven developer-demonstrators we visited were selected in 
consultation with Network officials to provide a range of experi- 
ences in terms* of* effectiveness and age of programs. 
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ANALYSIS J2F JE&E READINC^ ACHIEVEMENT 
GAINS OF TITLE I STUDENTS 




gURPOSE -OF ANALYSIS 

——W^T^—* , • . * ' 

We .analyzed -title r achievement* test sfaores^ in 14 school 
districts to .(1) examine tjhe^fef f ectiveness of the ti^J.e I projects 
in meeting the needs of educationally disadvantaged studeijts^ 
(•2) ^assess th6 needs, for project , improvement, and (3*) compare * > 
locar project results with reported exemplary project results to 
ascertain whether tl|e potentiaroenef its of adopting exemplary, 
prcj^jects^would be significant. ^ * 

SAMPLE 'sl^^jllON 

Fronuthe 47 scho61 districts chosen for site, visits (se,e 



app. I),^we selected, 13 districts fo? detailed- analysis of j^itle I 
project '1/ achievement* test* results.^ ,[ To as f sure balanced represen- 
tation., considered school district sj.ze, geographical location, 
and degree of urbanization* We also ^considered the availability 
, and quality of Achievement test d,afta. y To 'improve the geographical 
^balance, the sample of 13 districts was augmented , by adding a dis- , 
I trict not included in tfye original. 47 Our 'analysis of the 14 dis- 
tricts included about 16,500 students receiving title I services! 
in 'approximately 340 schools^ ^ , ' ^ 

Th^ title I sch^ls^we^Scn^lyzed were dispersed among the 
Nation's four geographic' regions', 2/ as follows: 

* » . "* * 
— N orthea&fct — 96— scho ols, 5 , '8 8 7' stud e nt s t e st ed^ t 

j — Southeast: 14 schools, 349 students tested. ^ 



--Central: 46 schopls," i*,494 students Jfested. 

— Western: . 184 schopls, 8,793 students tested. 

Seven of the school districts were "in large cities ha^i^g» over 
"100,000 population, four were in smaller cities having between 
10,000 and 100,000 nopulation, and three were in rural areas 
Four of the district had eftrollraents, of over, 50, 000 students, 
. seven had enrollments « of 8,000 to 50 < 000 students, and three ha4 
•enrollments of under 8,000 students. Eight districts* were in 

— ■ — : — s • 

jY l/ln some*' school districts, *title I project funds are supple- 

. mented wi testate and locat funds . * * * 

* * » 

2/,We 'used the satme regions defined in the National Assessment 

of Educational Progress. . # V 
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counties having a low-income population* o£ less tfian 17 percent, / 
four were in counties having low-income populations between* * . 
-16.7 and 22 .3, percent, and two were iri counties where more than 
56 percent of the population was low income. i 

ANALYSIS METHODOLOGY • 

Using existing achievement test scores for individual students, 

we tracked the progress of 22 student sample groups in reading 

achievement for 3 to 4 years,, depending .upon the availability of 

recordg in each district '. All 22 student samples were selected 

•from grades emphasized in the districts 1 title I programs. In' 

'some cases,. w<e traced the students 1 progress over a period of 

yearns, into grade levels, that d4d not provide title I services. 

All samples consisted of 100 percent of the students receiving 

title Y reading assistance im the grade levels selected for inir 

tial analysis. Students joining these groups in higher grades 

during subsequent years of title I assistance were added to the 

sample from their time of entry into the project. For example, 

if' the base sample consisted of aj.1 third grade title I reading 

students in school year 1975-76, a fourth grade student enrolling 

in title I~*sading in school year 1976-77 would be included in the 

"sample from the time of enrollment. These additions were necessary 

to show projeatyres^lts basecj on all students served, gather than 

on repeat students only* 

♦ * * 

To* determine the needyfor improvement *in local title I proj- 
ftcts, the following gainsffrere computed for individual groups of 
t students and their reepjptive .projects T ~ * 



f * 



— In-program g^, 
while repel v 



achieved by students du ring a sc hool yeaj 
title I assistance* 



-Continuing in-program gains achieved by students over con- 
sebutive years of ti.tle 1 service. > 

-Sustained gain rates mainji^ned by students after tljeir 
title K l assistance had b£en discontinued. 




^ The achievement^ gain ratel were analyzed to determine 
/(l) whether tlie, studerTts^were gaining or losing ground on their i 

riormal achieving peers iriefcelx^chool year, (2)>the cumulative 
V^£fect of consecutive years o£ltIti^^_ass instance on student 
rate% of achievement gain, and (3) the eTfectr-Hsf^i^continuing 
title I assistance on student gain rates. 

Achievement ga^tn measurement , * ' ^ 

Student achievement gains were measured by subtracting the 1 
pretest from the posttest achievement level for the period 
.... ^analyzed* Expanded, 0_tandarA_Xscale)... scores were used. , Average 
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gains for the projects were determined by aggregating and averag- 
ing the scores of students' served/ again using>standard (scale) 
scores. * Expected or nolrmal gain was defined as the amount of 
gain that the average achieving studejit at the same age -and grade 
level j/ould achieve for £he period of instruction. 

Where school districts used te^ts prepared by different pub- 
lishers within the several testing periods, we converted all te'st 
scores to .expanded standard scores of the t,est battery used, most 
frequently. This was accomplished using an* intermediate metric or 
scale, common to the test scores being, converted and the selected 
battery scores"; f The broadest category; of reading scores (total 
reading) rather than subtest scores (e.g., vocabulary and reading 
comprehension) were used. v • * ' — 

Limitations ' 4 

' 

The selected sample of 14 school districts is not large enough 
to enable us to project with a high degree of confidence that the 
"results typify title I pro ejects . nationwide. However, puj sample 
was carefully selected te 'a&sure a broad range of school district 
§uzes and environments, anctv^Jpelieve the'resjults of our analysis 
should be indicative *of many of the Nation's school district^. 

* In the districts we analyzed, we reviewed the test data, 
checked score conversions, and considered the adequacy of such 
factors as norming periods of the tests. Because we used test 
data available in the school districts and could not test select- 
ing, administering, and scoring, we cannot guarantee the total 
accuracy of the data. 

The derived test results may reflect some achievement gain 
inflation due to gain scd£e analysis and a statistical phenomenon 
Known as "regression toward the mean, 11 wherein students scoring 
well "below the pretest mean tend to make artificially high gains 
from pretest to posttest when the same test score is used for 
student selection and pretesting.* We believe, however, that the 
possible gain inflation j3ue to this phenomenon was reasonably 
limited. in our analysis because we followed the progress of stu- 
dents over several years and the same test score was generally 
not* used for initial' student selection and pretest measurement* 
In addition, inTour opinion, testing was usually done within a 
reasonably close time to the test publisher's norming period. * , 

expressed concern about our study 's^ reliance dja the grade 
equivalent metric or scale since it may be misinterpreted and can 
distort the measurement of achievement if misused. According to 
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a report prepared for the National Institute of Education 1/; on 
controversies in evaluating compensatory education, a basic ^problem 
in selecting a metric or scale is "* *~* one of validity versus 
communicaJbility; the more technically correct units are not neces- 
sarily .those that are easiiest to understand or directly relevant 
tp decisionmaking." Thja report claims that all available nfetrics 
gr scales have weaknesses in validity, cbmmunicability, or both. 
However, it states that ndne of the other metrics or scales has * 
the same clarity % and "simplicity of meaning as' grade equivalents. 
Accprdingly, we reported achievempat N rasults in grade equivalents 
,but used the more technically tori rect>xpanded .scale score to, 
measure academic achievement. J 



l/ M Controversies in the Evaluatibn of Compensatory Education, 
~~ American Institutes for Research, July 1977. , 
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UNITED STATES DEPARTMENT OF EDUCATION 

WASHINGTON. D.C. 30302 



APR 21 £8i 



~ ASSISTANT SECRETARY. 
FOR ELEMENTARY AND SECONDARY EDUCATION 



Mr. Gregory J. Ahart , • 
Director , Human Resources Division 

United States General Accounting' Office * * 

Washington, D.C. 20548 

^ * * 

Dear Mr. Ahart: 

* 

The Secretary, asked that I respond to your request for comments on your 
Draft Report entitled, "Greater Use of Exemplary Education Programs Could 
Help Improve Education for Disadvantaged Children." 

The enclose comments represent the tentative pdsition of the, Department and 
are subject to reevaluation when the final version of the Report is received. 

While we concur pith the thrust of all but one of the Report's recommendations, 
we wish to reiterate our concern about the study* 8 reliance on a grade- 
equivalent metric, since it is a metric often misinterpreted. In fact, as 
stated on page 40 of the Report, GA0 is aware thai grade-equivalent scores 
are not appropriate for determining average achievement levels. Therefore, 
it /is unfortunate .that this metric was used/-, 

We appreciate the opportunity to comment on this Draft Report before its 
publication* 



Sincerely, 




1 4 ' J < 4 

Sodrig^ek H j \ 
Lssistant Secr«ary J 



H. 

Acting Assistant 



Enclosure 



/ 



/ 



/ 



GAO note: 



r 

Page references in this appendix^ntay not correspond to* 
page* nurtfcfers- in the -final r^portv - - - - - - — - - : - 
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Comments o^the Department of Education on the Comptrollet^Gene^^s Proposed 
Draft R^rt to the Congress Erftitled, "Creates Use of. Exemplary Education 
Pr;fegriM Could Help Improve Education for Disadvantaged Children*** 

'. v . - : • 

/GAD Recommendation 

• , • 

To improve the procedures, for identifying and correcting Title* I project 
weaknesses, the Secretary of Education should prescribe. specif ic criteria for 
State and local school officials to use in assessing the effectiveness of 
Title I projects * . 

* - » 

Department Comment ^ 

We do not concur. The report claims that the Department has failed to provide 
States with adequate criteria for assessing Title I project quality and effec- 
tiveness, as required by law. This position is based upon the language in 
Sect ioif 151(f) of 'the Ehication Amendments* of 1574, recumbered Section "153(f). 2 
of the Bducation-Amendments of 1978, 20 H.S.C. 2832, which reads: 

** SPECIFICATION OF OBJECTIVE CRITERIA. - 
* The modeis developed by the Commissioner . 
shall specify objective criteria which 
shall Be utilized in the evaluation of \ 
' all programs and shall outline techniques ' 
(such as longitudinal studies of children 
.involved in such programs) and methodology 
(such as the use of tests which yield com- 
parable results) for producing data wfcL'ch 
are comparable on a statewida and nationwide 
basis •** (emphasis supplied) * 

Our position is that the above-cited provision requires the Department to 
provide criteria or standards to assist the school districts in choosing methods 
by which fhey can evaluate their Title I project*. We feel that the fina l 
regulations on Title I evaluation procedures, issued b? the Department on 
October 12, 1979, fully satisfy this statutory 'requirement. Those regulations 
include provisions on technical standards' to be used by lockl educational agencies 
(IZAs) in evaluating theijr Title I projects, and provisions .on the use .of evaluation 
models / 
• . u * 

GM> K however, feads the above-cited "provision to require the Department to pro- 
vide* criteria for assessing the quality and effectiveness of Title I projects. 
We have researched the legislative history of the Title I evaluation provisions 
and found, contrary to GA0*s' assertion, no indication that Congress intended tt^e 
Departaent to provide these kinds of criteria.' " 

The Report goes on to criticize the Department for failing to provide criteria for 
determining what levels of performance ^^satisfactory and what levels are in- ' 
■adequate* Aside from the fact that there is no legal' requirement for providing 
these criteria, to do so would conflict with the Departments, longstanding policy 
on non-intervention in the programmatic decisionmaking process at the State and 
local levels. In fact, Section 432 of the General Education Provisions Act 
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prohibits the Department from Exercising any direction or control over the 
programming and administration in the, State and. local agencies. The Department 
recognizes that different school districts have .different needs ^concerns, 
and therefore feels that such questions as "What is the satisfactory level of / 
performance?" and "What is an effective Title I program?" are be«i answered by / 
the State educational agencies~(SEAs) and LEAs most aware of those needs and / 
concerns. ._ 

ED ha? coventrated its activities, on assisting SEAs and LEAs to conduct valid 
and ultimately usable evaluations of .Title I projects. Our strategy has been 
evolutionary. Bearing in mind that the majority of Title I districts are quite 
small and probably do not have trained evaluators , • we have concentrated t our re- 
sources on fundamentals—identifying appropriate tests; administering, scoring, 
and analyzing the results accurately; correctly implementing an evaluation 
model; reviewing and interpreting the results; and, finally, making judgments about 
project components. After implementing the Title I Evaluation and Reporting System 
(TIERS) for several 'years, many SEAs and TJ^As av$, beginning to monitor closely the 
•patterns of results JifXL their Title I evaluations. They are: (1) developing 
monitoring •plans to revTew the particularly ^\%essful and- unsuccessful pr^ects, 
(2) a&king questions relating to the implet&nta^ion of projects that -are extreme, - 
and (3) beginning to Identify strengths and IweMKnesses of their projects. We will 
provide technical assistance, as requested, t6 conduct process evaluations of un- ♦ 
usually effective or ineffective projects, aidd to assist in identifying and dis-. „ 
seminating effective local practices and identifying and correcting program weak- 
nesses. Implementation of the TIERS will permit LEAs to assess Title I achievement 
gains of each Title I school and SEAs to review results across LEAs since the re- 
porting metric will be' uniform and the results comparable. ED f s national aggregation 
will enable SEAs and LEAs to view the levels of their gains in light of the ranges of, 
gains reported nationally. 

The strategy ED has adopted^ does produce an interpretable measure of whether Title I 
projects are effective. Interpretation of results, however, should and does differ 
from site to site. The Title I evaluation models yield measures of achievement gain, 
i.e., how much Title I students gaiii above what would have* beett expected in the ab- 
sence of Title I. A positive gain/from an evaluation, shows that a project has sue- 
cessfully increased the rate of growth of ita participants. In theory, the larger the 
gain, the Jitter the' program. A /negative ^gain" indicates that the Title I partici- 
pants are growing at a slower ta£e than their non-Title I, but similarly disadvantaged, 
SL, Specification of criteria to separate the satisfactory projects from the un- 
satisfactory projects is technically unsound, educationally indefensible, and an 
intrusion into the legal auth^tity of the States and their local school districts. 

GAP Recommendation 

T o Improve the procedures for identifying and correcting Title I project weak- 
nesses ♦ the Secretary of Education should expand ED f s monitoring efforts for 
assessing whether State education agencies are (1) meeting their responsibilities 
f rt OT «1tiAtP the effectiveness of local Title I projects, and (2) assisting local 



to evaluate the effectiveness of local Title I projects 
school districts to improve their Title I projects 
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Department Comment s ^ - 

We concurl ED places a high value on the Stale's responsibility to evaluate the 
'effectiveness of local Title I projects and to provide assistance to Ibcal school 
districts to improve their Title I projects. 

ED teams' review annually each State's administration of Title I, assessing how well 
States are meeting theit evaluation responsibilities and recommending actions to 
be taken by the SEA to improve its overall administration of Title I. ^ 

In preparing for their onsite monitoring visits the .ED Review teams examine State 
evaluation reports and a sample of local Agency reports tor compliance with 
evaluation requirements and to assess program qual|ty><They also review Technical 
Assistance Center (TAC) reports which often suggest evaluation improvements to be 
"made by the State. These are discussed with program and evaluation specialists 
in the SEA for the purpose of outlining team onsite strategies^ 

While visiting State agencies, the teams review evaluation' practices, including 
technical assistance provided- and the use made of local evaluations during the 
applicatibn preparation and approval process., At local -agencies they review 
evaluation practices and discuss the use, of evaluation results as a basis for * 
program improvements. 

On October 12, 1979, the final Title I evaluation regulations were published. 
ED review teams were thus able to assess State and local adherence to^ the standards 
ard i&dels contained therein. During FY 1980 ED reviews devoted greater attention 
than in past years to the area, of Title I evaluation and their reports to the - 
States reflect this increased attention. ED * intends to continue this efprt to 
insure that J3EAs and LEAs meet Sitle X evaluation requirements as well /s to - 
improve the quality and use of evaluations at State and local levels. aJso, for 
the fifth year now, the Title I TACs are assisting SEAs and their LEAs in imr - 
proviso the quality of evaluations of Title I projects and in "using evaluation 
results for program improvement. This assistance takes the form of materials 
development related to needs assessment, student selection, testing, and 
product and_process evaluation. In Response to requests from SEAs and LEAs, 
workshops related to test selection* evaluation design, needs assessment, instruc- 
tional decisionmaking ^and evaluation are developed and presented. 

The TAC .services* are provided free of charge toWate end local education personnel 
to assist on a variety of Tttle Revaluation issues. As a result of .the TAC services 
and tfe development of the report ii^i system, State and local education personnel 
are conducting more complete 'evaluations of their programs. These evaluations are 
of a continued higher quality because the system stresses evaluation data quality 
control and the TACs provide technical assistance to support this effort. The 
development of sustained gains evaluation at the local level is providing 'local 
personnel with data about the effect xfc Title I over a longer period of time than_ 
one year. SEAs and LEAs are .devoting increasing attention to- the area of process 
evaluation! -This phase-^f evaluation provides contextual input into the instruc w 
tional decision process. .With this increased capacity, State and local education 
personnel are able 'to evaluate programs mott* effectively and use these data for 
program, development. 
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>GAO Recogaendation^ 




To igprove the proceduigs for identifying and correcting Ti tle I project weaknesses, 
the Secretary-^ Education should direct State education ag encies to (1) asses? the 
validity of procedures used by local school districts to evalu ate their Title I 
projects, (rfS prescribe corrective measures where evaluation procedures are 
Inadequate, and (3) assist local school officials in the u se of evaluations for 
detection and correction of project weaknesses. « 

Department Comment , 

We concur. As indicated in our res pons a to thp monitoring recommendation, ED teams 
review St^te ltdmlnistratien of Title I,^inciuding site visits to a number of EEAfl / , 
Jach year to observe local project operation^ As appropriate, recommendations are 
4ade for SEA improvement in the area of evaluation. 

In compliance wittuSection 151 of Public Law 93-380, the Education Amendments pf 
1974, ED developed evaluation models and standards for use by- SEAs and LEAs. On 
October llf 1979, final regulations were published specifying models^for evaluating 
the effectiveness of LEA projects providing instructipnal services ^reading, 
language- arts, or mathematics. However, as early as 1978, although not yet federally 
mandated, approximately 5,500 school districts were in the process of implementing 
one of the models. The experiences of ED, the SEAs, and the LEAs in u*ing these 
»dels were disseminated in a pamphlet entitled, "The U.S. Office of Education Models 
to Evaluate E.S.E.A. Title I: Experiences After One Year of Use.". 

In addition, £he ten regional TACa prepare support materials to use. inlronj unction 
with their workshops and personal (onjite, telephone, written) consultations. 
Materials produced include .topical papers (e.g., out-of-level testing); simulation 
exercises (e.g., test selection reporting forms completion); checklists for 
Implementing a particular model; handouts (e.g., test charts); and transparencies 
for use with an overhead projector. *. . - - / t 

- , ■ / 

The TACs keep the States aware of the availability of technical distance through 
Regional Coordinating Council meetings, scheduled meetings with Sttete contacts on 
an individual^ or group basis, and distribution of regional ^newsletters. 

Any SEA or LEA may request services from ED or directly Jhrdm ltd TAC Assistance 
is^often provided as a follow-up to ED review team filings and/ recommendations. 
ED will continue to support efforts in these areas.' 



-GAO-Recoaaendatloti 

To lapr ove the procedures, for identifying and correcting Titje I project weaknesses 



the Secretary of Education should advise State education ag 
school officials to' use the assistance available through ~*~ 



Ineffective and inefficient Title I projecrtsT 




cies to encourage local 
Network for Improving 



ERLC- 



52 



63 



APPENPIX III 



\ 



APPENDIX III 



Department Comment , ^ 

We concur. "Exemplary title I programs disseminated by the National Diffusion . 
Network (fflfe) should be included in each Stated repertoire of strategies for 
improving teaching and learning in Title I schools. Chief- State School Officers, 
State Title I Coordinators, and State Facilitators should encourage Title I 
districts to examine exemplary Title I programs diffused by the NDN for possible 
solutions to their school improvement problems. Base v d upon previous studies, 
this encouragement should provide school .^strict representatives with opportuni- 
ties to meet face-to-face with the developers of many ' exemplary programs. This 
process gives district representatives opportunities to- examine materials and ask 
questions about evaluation results? installation costs, program philosophy, and 
graining requirements. ED plans to continue its efforts to^ encourage SEA% to work 
with I£As to assess the*, effectiveness of local Title I programs, to improve 
local projects, as needed, through the adoption of exemplary projects, and to 
encourage LEAfl* to critique their own results and to consider the adoption of an 
exemplary project /as appropriate. Encouragement and guidance is provided to SEAs 
and LEAs through ED t s annual national dissemination meetings, ED review team 
©anitoting, and materials developed and distributed nationally. After examining 
several alternatives, the district can choose the program that best fit^s its needs 
and resources and 'arrange for staff training. Of course, not all districts**lrill 
find programs in the NDN to fit their needs and may have to turn to other sources. 



6A0 Recommendation 



To convince potential adopters of the merits of exemplary 



he ^Secretary 



of Education should direct the Network to provide complete, comparab le, ana current 
i nformation about the capabilities of the Network's exemplary proje cts, including 
"CD their 1 impact on scholastic achievement, student behavior and sustained growth, 
(2) their costs, and (3) the results of past adoptions by other schools. 

Department Comment ' 

We concur. A brief summary of this information is included in the program 
description in Educational Programs That Work , a publication issued annually by the 
NDR program. Also included in the publication is the name and telephone mmber of 
the project director. A telephone call or letter can be used to obtain additional 
information about a program. In addition, all oi tfiis information can be obtained 
face-to-face from the developer of an exemplary program' during ED or SEA-spqnsored/ 
awareness conferences. The SRI International evaluation of NDN found that most 1 
decisions to install an NDN exemplary program are made as a result represents- i 
tives from a school district either visiting the program to see it in operation, 6r 
attending an awareness conference and, talking with the* developer of the program. 
We will continue to promote national and State level awareness conferences *so thit 
developers and interested LEA representatives can meet -face-to-face. ■ 
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As re^ui^ed by .the EDN regulations^ by>9§4 for each exemplary project to retain 
its eramjjlary status, project dawctfst be submitted to trie Joint Dissemination 
Revic&T Panel (JDRP) doaimenting^fcflat* the results are still outstanding, that 'there 
have/been adoptions of the^ppprject, and that the results achieved at the adopting 
. sites are also outstanding-rUse of the Title I evaluation models by all LEAs will 
result in JDRP submissions with uniform, comparable evidence of achievement imnact. 
The Vgrmat ^j^etgSiSissions to the JDRP, both for new projects .and for thos6Vbsing 
revaii3ated?i8 established by the JDRP and contains a specific format for reporting 
data on project start-up and operating costs* 

The Title I statute (P.L*/f 5-561y£ection 124(g)) requires each LEA t$ examine 
daring each thre^e-year period the sustained effects of Title I services* ED has 
developed and the TACs are disseminating altertiativesj for JLEAs _to elhgloy t^o .meet 
this requirement and to use the results for project improvement* Although 
^staining achievement gains over* time is one aspect of project success to be 
considered In a review of the 'impact of an exemplary project, the methodological^ *. 
problems »in conducting a longitudinal stuldy are by no means trivial* However, 
some evidence of sustained effects will be available from each exemplary project 
as part of its resuWission to the JDRP*. • : 

Oujr experience with the NDN indicates that few, decisions to install a program were 
made as a result of reading printed Bteterials* Developer-Demonstrator^ project 
directors Report that very few questions are ask^d by ^potential ^adopters about 
evaluation results obtained. We concur that every exemplary project should make * 
this information available upon request, and we believe that the one^page descrip- 
tions in Educational Programs That Work can be improved to provide more information. 
However, if school officials in 41 out of the 1*7 school districts visited (87%) were 
satisfied with their Title I programs (page 36), we doubt -that the publication of * 
Ttf>re< elaborate and>costly information about programs will result in major increases 
in the number of adaptions* Data on the effectiveness of t*he exemplary .programs 
were available/ but school officials apparently chose not to- make^inquiries. We do 
believe that encouraging sctool officials in the six districts outj of* forty-seven 
that were not satisfied wit67their programs to attend awareness conferences will 
increase the current rate of Title I adoptions. 1BD will continue to wdrJ&Nwith' SEAs 
to stimulate interest on the part of LEAs to analyze results of their currbnt -proj- 
^ects^and to consider tjie merit of adopting an exemplary project, if appropriate. 

,GAO Recomegdation * 

To provide more accurate data on the volume of Title I adoptions of exemplary proj- 
ects and insure that adoptions of the projects are successful, the Secretary of 
Education should make sure that the Network's developer-demonstrators followups o f 
project adoptions are successful in % * 

— providing accurate data on the number of Title I adoptions and the 
schools affected, 

— determining whether the projects have^ been installed correctly and'are 
, functioning properly, and 

~ providing additional implementation assistance where needed. 
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Department Comment - * v, 

We concur. &)N does collect accurate data on* the number oHadopt ions', the number 
of Schools „ *nd the number of teachers and students participating in the adpptions 
during .the reporting period. It does not collect data on which adoptions are 

# Jitl§ I projects or on the°number of Title I schools affected. When the present 
reporting fofcas were cleared, pressures to reduce paperwork and reporting require-, 
aents prevented NDN from collecting information on the numerous sources of funds, 
including Title I funds, used to install NDN programs. We v do intend to request 
from the Office of Management and Budget clearance .to collect information on the 
particular funding source used to adopt an exemplary project. 

We recognize as well the value of involving the Developer-Demonstrator (DD) as an 
active participant in the adoption process and follow-up. Using existing resources, 
ED plana to make greater use of the DDs to monitor, claimed adoptions in each State/ 
* to gather information on adoptions (including achievement 4 results), and to maintain 
contact with adopting sites over ti>me. Presently, DDs maintain contact with/their 
"adopting slteS by teleptsone and, if within the DD f s State, by on-site visits. When 
^resubmitting forTiDRP approval prior to 1984, each DD must include achievement data 
trtfcTlive >ad0ptiig sites reflecting the diversity of sites which have adopted t£e- 
project.* ^liie c^rent funding leyel (approximately $40,000) of each DD is not 
suffictenVto* support all activities in 50 States for which Developer-Demonstrators 
could* be responsible, 'such as: N "* ( \ 

* * 
° participating in awareness conferences 7 so 'that potential adopters 
, can '-obtaib information" directly 1 from t)ie developer; * •* 

'.V- conducting" training for.- personnel in districts' that twfe made 
commitments to install their program; 

* ° monitoring the program after it hasbedn installed; and 

* * °* providing follow-up technical assistance where needed.. 

NDN is attempting to provide these services within its funding limitations. 
£ Much more emphasis 16 being placed upon the quality of implementation_and the , 

* persistence* of installations than was the case in the early years of NDN. The 1 * 
number of Developer-Demonstrator Projects funded by NDN As being reduced so that 
the average level of ftmding carf'be increased -slightly to make additional resources 
available for monitoring 'and foltow-up implementation assistance where needed. In 
the future, ED will require more jtnforaation from each DD on, the status of adoptions* 
Thus* the linkage between DD and adopters ^will be strengthened and maintained. 
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